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The  MODIA  system  (Method  of  Designing  instructional  Alternatives)  is  a 
computerlied  planning  tool  initially  designed  by  Rand  Corporation  for 
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research  pur-; *2$  and  subsequently  examined  in  Air  Training  Command  for 
use  in  facilitating  course  planning.  The  MODIA  system  was  evaluated  at 
Keesler  Technical  Training  Center  to  determine  its  usefulness  to  branch, 
group,  and  center-level  course  managers.  Additional  objectives  involved 
the  assessment  of  changes  needed  to  improve  MODIA  effectiveness,  the 
feasibility  of  implementing  MODIA  at  Keesler  in  the  immediate  future 
and  the  possibility  of  different  management  levels  using  various  parts 
of  the  MODIA  system  separately.  The  results  of  the  Phase  II  Evaluation 
were  also  compared  with  results  obtained  in  a Phase  I evaluation. 

Seven  courses  at  Keesler  were  initially  identified  for  simulation  and 
costing  using  the  MODIA  system  in  the  Phase  II  service  test.  Two  groups 
of  three  courses  each  were  to  be  planned  under  the  family  grouping  concept 
to  demonstrate  sharing  of  technical  training  resources  between  related 
courses.  An  additional  course  was  selected  because  of  its  length  and  number 
of  resources.  ISD  personnel  from  Keesler  Technical  Training  Groups  provided 
a MODIA  interface  team  with  course  data  to  input  into  the  MODIA  system. 
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The  resultant  products  were  a simulation  of  the  course  operation  and  a 
detailed  listing  of  course  costs  associated  with  a given  course  design. 

Costs  and  simulations  of  alternative  designs  were  also  generated.  These 
products,  including  alternative  designs,  were  given  to  branch  and  group- 
level  course  managers  who  were  asked  to  judge  the  utility,  to  them,  of 
the  information  generated  by  MODIA  in  managing  and  planning  the  selected 
courses. 

This  service  test  resulted  in  simulations  of  five  courses  at  Keesler 
and  a number  of  alternative  course  designs.  Cost  information  on  basic 
course  and  alternative  course  designs  were  provided  course  managers  who 
judged  the  products  as  being  of  limited  value  to  them  in  managing  course 
operations  more  effectively.  The  limitations  of  the  simulation  were  of 
such  a nature  as  to  probably  prevent  the  present  system  from  being  easily 
changed  to  make  it  acceptable  to  planners  and  managers.  Personnel  involved 
in  the  service  test  generally  liked  the  concept  of  simulation  of  alternative 
course  designs  and  the  costing  of  alternative  designs,  but  felt  the  present 
MODIA  system  was  too  limited.  Additionally,  the  present  MODIA  system  could 
not  be  implemented  at  Keesler  due  to  the  demands  placed  by  the  system  on 
current  computer  resources. 

RECOMMENDATION 

Recommend  the  MODIA  System  not  be  adopted  for  use  in  ATC  at 
this  time. 
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PHASE  II  EVALUATION  OF  THE  MODIA  SYSTEM 
SECTION  I - INTRODUCTION 

1.  Authority:  This  Phase  II  service  test  and  evaluation  of  the  MODIA 
(Method  of  Designing  Instructional  Alternatives)  planning  system  was 
conducted  under  the  authority  of  HQ  ATC/XP  letter,  16  Dec  1976,  Phase  II 
Evaluation  of  MODIA,  and  the  Plan  for  Evaluating  the  MODIA  System, 

14  Oct  1977. 

2.  Background:  Rand  Corporation  initially  designed  the  MODIA  system 
as  a research  tool.  Air  Training  Command  has  examined  the  potential 

I 

of  the  system  for  use  in  facilitating  course  planning.  The  primary 

objective  of  the  MODIA  system  is  to  provide  a systematic  process  of 

relating  quantitative  resource  requirements  to  course  design  and 

operation.  MODIA  should  enable  planners  to  consider  different  sequences 

of  course  objectives,  alternative  sequences  of  subject  matter,  varying 

teaching  methods  and  types  of  instruction,  and  different  mixes  of 

students,  equipment,  and  facilities.  Moreover,  MODIA  should  simulate 

the  way  in  which  students  progress  through  alternative  course  designs. 

The  MODIA  planning  system  has  four  components:  the  description  of 

options  for  course  design,  the  User  Interface  (UI),  the  Resource 

Utilization  Model  (RUM),  and  the  Cost  Model  (MODCOM).  The  UI  is  the 

Interactive  portion  of  MODIA,  the  RUM  provides  the  simulation  of 

123 

course  operation  and  the  MODCOM  provides  course  costs. 

a.  The  initial  development  of  MODIA  was  completed  in  October  of  1973. 
ISO  teams  from  Keesler  and  Lowry  performed  a critical  design  review  at  that 
time.  Rand  Corporation  made  several  revisions  based  on  the  design  review, 
and  the  Phase  I service  test  of  MODIA  was  conducted  at  Keesler  AFB  from 
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March  1976  to  June  1976.  The  results  of  the  service  test  were  reported 

4 

in  ATC  Project  Report  76-1  (30  July  1976).  The  results  generally  indicated 
that  MODIA,  “has  the  potential  to  be  an  effective  planning  tool  whose  use 
could  lead  to  more  cost-effective  technical  training  courses."  Several 
important  questions,  however,  could  not  be  addressed  in  the  Phase  I evalu- 
ation. This  study  provides  data  relevant  to  those  unanswered  questions. 

b.  During  Phase  I,  Rand  personnel  reached  the  conclusion  that  the 
MODIA  system  was  too  complicated  to  be  used  effectively  by  the  planners 
themselves,  and  as  a result,  a team  of  individuals  was  trained  in  the 
operation  of  MODIA.  This  group,  subsequently  called  the  interface  team, 
operated  MODIA  while  the  course  planners  provided  the  planning  data  needed 
to  design  courses.  The  concept  of  the  interface  team  carried  over  in  this 
evaluation. 

c.  The  physical  arrangements  at  Keesler  during  Phase  II  were  very 
similar  to  those  arrangements  which  existed  during  Phase  I.  The  special 
features  of  these  arrangements  included:  (1)  a room  in  which  the  interface 
team  operated  a remote  terminal;  (2)  a Class  A telephone  line  used  with  an 
acoustical  coupler;  (3)  a MODEM  (Bell  103A  Data  Set)  in  the  computer 
facilities  connected  on  a dedicated  line  to  Biloxi,  Mississippi  telephone 
exchange;  (4)  one  of  the  primary  provisions  of  both  Phase  I and  Phase  II 
tests  was  that  they  be  conducted  on  a "non-interference"  basis.  During 

the  shakedown  of  the  MODIA  system  (prior  to  the  actual  start  of  the  Phase  II 
service  test),  an  attempt  was  made  to  use  MODIA  to  set  up  a rather  lengthy 
course  and  it  was  determined  that  there  could  be  a serious  interference  prob- 
lem with  on-going  training  efforts  and  other  users  of  the  Honeywell  6060.  This 
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(2)  Assess  the  utility  of  the  system  to  the  technical  training  school 
management. 

(3)  Determine  MODIA's  usefulness  given  currently  existing  resource 
constraints,  current  computer  support  capability,  and  training  policy. 

c.  Determine  MODIA's  usefulness  as  a problem-solving  tool. 

d.  Determine  the  organizational  configuration  and  operational  procedures 
which  may  be  used  in  applying  MODIA  at  Keesler. 

e.  Determine  resources  required  to  implement  MODIA  at  Keesler  in  the 
immediate  future. 

f.  Determine  what  changes  to  MODIA  are  needed  to  improve  its  effectiveness. 

g.  Examine  the  potential  for  using  the  cost  model  (MODCOM)  as  a stand- 
alone system. 

h.  Develop  a training  program  for  the  use  of  MODIA  - including  develop- 
ment of  a "User  Interface  Team  Guide". 

SECTION  II  - METHOD 

4.  ISP  Team  Make-up:  Initially,  it  was  planned  that  each  ISD  team  would 
be  composed  of  a curriculum  training  specialist,  a training  resource  specialist, 
and  a subject  matter  specialist.  In  practice,  however,  the  interface  team 
member  from  each  of  the  three  training  groups  involved  in  this  exercise 
worked  with  only  one  other  person  from  the  training  group.  This  person 
provided  the  primary  ISD  Input.  Others  were  Involved  as  needed  In  the 

I 

planning  of  different  parts  of  a given  course  (for  example,  the  planning 
of  3ABR32831  involved  up  to  as  many  as  5 ISD  people).  The  reorganization 
of  the  technical  training  center  and  shortage-  of  experienced  planners 
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dictated  this  deviation  from  the  evaluation  plan.  It  should  be  mentioned 
that  the  use  of  fewer  people  significantly  drove  down  the  personnel  cost 
of  planning  with  the  MODIA  system.  In  contrast  to  Phase  I cost  analysis 
which  included  personnel  costs  of  many  ISO  team  members,  this  service  test 
figures  personnel  costs  associated  with  only  one  or  at  most  several  (in  the 
case  of  3ABR32831)  ISO  team  members'  time.  Each  ISD  member  was  responsible 
for  revising  the  selected  courses  with  inputs  from  other  tech  school  person- 
nel as  needed. 

5.  The  MODIA  Interface  Team:  The  interface  team  was  composed  of  a GS-11 
(Mr.  T.  M.  Gol 1 day/TTGT I ) ; a Master  Segeant  (Michael  A.  Stone/TTKSR);  and 

a Technical  Sergeant  (Wayne  T.  Wolski/TTKN).  The  training  of  the  interface 
team  was  accomplished  at  the  Rand  Corporation,  Santa  Monica,  California, 
d^ing  the  period  12  Sep  77  to  25  Sep  77.  This  group  of  individuals  served 
as  the  interface  between  the  ISD  planner  and  the  MODIA  system. 

6.  Course  Selection:  Courses  selected  for  MODIA  service  test  during  Phase 
II  were: 

Training  Course 

Group  Number  _Title 

3380  TTG  3ABR32831  Avionics  Nav  System  Specialist 

3390  TTG  3ABR27630  A CAW  Systems  Operator  (Manual) 

3390  TTG  3ABR27630-001  A CAW  Systems  Operator  (sake) 

3390  TTG  3ABR27630-002  A CAW  Systems  Operator  (407L) 

3410  TTG  3ABR30434-1  Ground  Radio  Equipment  Repairman  (Titan) 

3410  TTG  3ABR30435-5  Ground  Radio  Equipment  Repairman  (Minuteman) 

3*10  TTG  3ABR30434  Ground  Radio  Equipment  Repairman 

Course  selection  was  based  on  the  following  conditions: 

(1)  There  were  two  sets  of  family  group  courses  (the  27630  and  the 
30434  courses)  that  had  to  be  revised. 

(2)  Low,  medium  and  high  student  loads  were  represented. 

(3)  Different  instructional  approaches  were  represented. 
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(4)  All  courses  were  planned  through  all  5 steps  of  the  ISD 

process. 

(5)  One  of  the  courses  (32831)  was  of  long  duration  and  used 
a great  many  resources. 

The  assumption  underlying  this  course  selection  was  that  these  courses 
represented  the  best  planning  possible  by  conventional  means.  If  MODIA 
can  be  effectively  used  in  the  ISD  process,  then  both  planners  and 
managers  can  improve  course  designs  by  allowing  for  more  cost  effective 
planning. 

7.  MODIA  Service  Test  Costs  --  Data  Collection. 

a.  It  is  important  to  make  a distinction  here  between  Phase  I and 
Phase  II.  Phase  I results  showed  that  MODIA  could  be  used  by  training 
branch  and  group  level  personnel  to  decrease  course  costs  through  better 
design  if  they  could  use  any  design  they  chose,  regardless  of  training 
policy  or  personnel  management  considerations.  Phase  II  attempted  to 
see  how  well  they  could  use  MODIA  to  manage  costs  in  the  present  training 
environment,  and  within  real-world  constraints.  One  of  the  goals  of  the 
Phase  II  effort  was  to  examine  the  cost  of  the  MODIA  system  in  the  light 
of  such  restraints.  In  order  to  accomplish  this  goal,  all  elements  of 
the  system  cost  were  collected  as  outlined  in  the  evaluation  plan. 

b.  Specifical ly: 

(1)  Data  automation  cost  information  was  supplied  by  Keesler  ACD. 

(2)  Manpower  facilities  and  equipment  costs  were  collected  by 
KTTC/TTGH.  Facility  costs  were  not  counted  as  a cost  in  this  evaluation 
as  they  were  also  not  counted  in  Phase  I.  Also,  Rand  provided  the  TI-700 
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terminal  for  use  in  the  service  test.  This  terminal  cost  was  not 


counted  as  a cost  of  the  service  test;  however,  the  costs  of  terminal 
"time"  on  the  H-6060  computer  were  counted  as  part  of  the  data  auto- 
mation costs.  Total  equipment  and  manpower  cost  breakouts,  by  course,  are 
provided  at  Appendix  1.  These  costs  will  be  discussed  in  the  Results 
Section,  paragraph  17. 

(3)  EDP  and  Communications  Costs  were  provided  by  the  2052  Comm. 
Sq.,  Keesler  AFB  and  are  also  included  in  Appendix  1. 

c.  The  primary  cost  of  MODIA  was  in  the  manpower  and  computer  time 
required  to  support  the  system.  This  data  was  gathered  through  work  logs/ 
time  sheets  and  a terminal  log  kept  by  interface  team  members  and  ISD 
team  members  throughout  the  course  or  the  service  test.  The  work  logs/ 
time  sheets  were  filled  out  on  a weekly  basis  to  insure  current  and 
reasonable  estimates  of  time  spent  on  various  portions  of  the  MODIA 
service  test.  Course  cost  data  and  cost  information  for  use  as  inputs 
into  MODCOM  were  provided  by  the  Comptroller  and  from  Keesler  Production 
Analysi s. 

d.  Requirements  for  computer  resources  (CPU  time  and  time-sharing 
storage  requirements)  for  the  various  portions  of  the  User  Interface 
are  provided  at  Appendix  2.  However,  several  recommended  changes  to 
the  MODIA  programs  are  probably  extensive  enough  to  significantly 
alter  the  operating  characteristics  of  the  system,  should  a DAR  be 
developed  for  this  system. 
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e.  Specific  changes  to  the  MODIA  system  were  provided  in  written 
comments  by  1SD  personnel  and  all  interface  team  members,  as  well  as 
training  managers. 

f.  Down-time,  equipment  malfunctions  and  waiting  time  were  not  counted 
as  a direct  cost  of  MODIA  planning,  since  it  was  assumed  that  such  costs 
would  be  minimal  with  a fully  operational  MODIA  system. 

8.  Questionnaires: 

a.  Interface  team  questionnaires  with  raw  data  summaries  are 
provided  at  Appendix  3. 

b.  ISD  team  questionnaires  with  raw  data  summaries  are  provided 
at  Appendix  4. 

c.  Training  managers'  questionnaires  with  raw  data  summaries  are 
provided  at  Appendix  5. 

d.  Data  contained  in  responses  to  the  questions  in  all  questionnaires 
were  summarized  and  consolidated  to  provide  opinion  information  on  MODIA' s 
usefulness  as  a planning  tool  and  as  a problem-solving  tool.  Additionally, 
the  information  provided  from  questionnaire  comments  provided  a basis  for 
recommended  changes  to  the  MODIA  system. 


SECTION  II  - RESULTS 

9.  General : In  response  to  one  of  the  recommendations  in  ATC  PR  76-1, 
Evaluation  of  the  MODIA  System,  one  of  the  primary  purposes  of  this  evalu- 
ation was  to  determine  the  utility  of  the  MODIA  system  to  technical  training 
management.  In  fact,  the  Phase  I report  went  so  far  as  to  say  that  when 
procedural  questions  and  organizational  configuration  questions  were  resolved, 
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"It  appears  that  SAAS  management  will  be  able  to  show  that  MODIA  can 
improve  resource  management  in  a technical  training  environment  (para  lBg, 
p.  33,  ATC  PR  76-1)."  The  results  of  the  Phase  II  evaluation  dictate  a 
different  conclusion.  The  next  section  will  first  discuss  the  implementa- 
tion and  operating  costs  of  the  service  test  at  Keesler,  and  then  report 
the  results  as  they  relate  to  the  objectives  previously  outlined. 

10.  MODIA  Phase  II  Service  Test  Costs:  For  the  Phase  II  service  test 
implementation,  it  cost  Keesler  Technical  Training  Center  $11,074.  For 
the  operation  of  the  system  during  the  service  test,  it  cost  the  technical 
training  center  $44,297.  (see  Appendix  1 for  complete  cost  breakdown  by 
course).  These  figures  are  considerably  different  from  those  obtained  dur- 
ing the  Phase  1 evaluation.  For  example,  the  Phase  I report  placed  the 
implementation  cost  at  around  $36,000.  The  approximate  $25,000  difference 
between  that  service  test  and  this  one  can  be  explained  by  taking  into 
account  several  important  differences  between  the  two  service  tests.  These 
differences  include  a drastic  reduction  in  the  number  of  personnel  and  manhours 
involved  in  the  shakedown  and  set-up  phase  of  the  service  test.  Additionally, 
fewer  manhours  were  needed  to  supply  interface  team  members  with  planning 
factors  - i.e.,  there  were  fewer  people  involved  with  day-to-day  operation 
of  the  system.  More  about  these  implementation  and  operating  cost  differences 
will  be  mentioned  in  the  discussion  section.  The  results  of  this  portion  of 
the  evaluation  also  show  large  differences  in  the  cost  of  computer  operations 
when  compared  with  those  results  reported  in  the  Phase  I effort.  The  $1. 59/mi n 
reported  in  Phase  I is  considerably  less  than  the  $3.67/min  reported  in  this 
evaluation.  The  computer  costs,  as  well  as  program  specifications  and  operating 
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characteristics  of  the  M001A  software  are  addressed  in  the  following 
section  as  they  pertain  to  the  potential  data  base  for  the  development  of  a 
DAR  should  MODIA  be  adopted.  Overall,  the  total  service  test  cost  was  less 
than  anticipated,  despite  the  fact  that  computer  costs  were  substantially 
greater  than  those  costs  obtained  in  Phase  I. 

11.  MODIA  System  Operating  Characteristics  and  Limitations.  As  mentioned 
in  the  Introduction  Section,  the  size  of  the  UI  portion  of  the  MODIA  system 
was  so  large  that  it  caused  some  initial  interference  with  the  training 
mission  --  and  resulted  in  restricted  operating  hours  for  the  MODIA  service 
test.  The  primary  result  of  the  service  test  experience  that  pertains  to 
the  operating  characteristics  is  that  the  UI  (and  especially  the  "C"  phase) 
is  much  too  large.  While  it  is  conceivable  the  UI  could  be  made  smaller  and 
more  efficient,  serious  consideration  would  have  to  be  given  to  size  trade-offs 
involved  with  the  RUM  program.  This  trade-off  is  discussed  in  para  15  of  the 
Results  section. 

a.  In  general,  the  operating  problems  experienced  during  the  service 
test  can  be  traced  to  the  fact  that  MODIA  programs  were  written  for  an  IBM 
370/158,  and  were  somewhat  incompatible  with  the  Honeywell  6060  system.  Another 
important  factor  in  the  incompatibility  with  the  IBM  system  is  a virtual  storage 
system,  while  the  Honeywell  system  is  a segmented  storage  system  which  uses 
program  overlays.  Aside  from  some  basic  incompatibility  between  the  MODIA 
software  and  the  Honeywell  system,  there  were  other  problems  encountered  in 
using  the  UI. 

b.  Several  specific  findings  regarding  the  operation  and  use  of  the  MODIA 
programs  were  garnered  from  interface  team  members'  responses  in  their  ques- 
tionnaire. Most  team  members  found  it  easy  to  assign  learning  objectives 
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(course  content)  on  the  UI  with  some  notable  exceptions.  First,  MODIA  was 
unable  to  handle  assignment  of  course  content  under  the  family  grouping  concept. 
The  250  learning  event  limit  was  much  too  restrictive  for  planners  to  use  MODIA 
for  simulating  courses  with  shared  resources.  Two  of  the  basic  courses  planned 
in  this  exercise  did  have  221  and  241  learning  events  each.  Basic,  single 
courses  could  easily  fit  within  the  limitations.  However,  courses  planned 
under  family  grouping  required  up  to  three  times  the  250  learning  event  limit. 
Second,  garbling  prevented  the  assignment  of  lesson  objectives  which  had 
certain  letter/number  combinations.  For  example,  in  the  3ABR  30434  course,  . 
the  interface  team  member  entered  subl0309  for  a learning  objective,  but  the 
computer  read  1DANTCU.  This  garbling  was  a factor  throughout  the  service 
test.  Nevertheless,  from  responses  to  the  questionnaire,  it  appeared  that  the 
interface  team  was  sufficiently  trained  to  be  able  to  handle  most  problems 
that  arose  in  assigning  course  content. 

c.  The  assignment  of  resources  to  specific  learning  events,  however,  was 
a different  matter.  In  all  cases,  interface  team  members  found  it  difficult, 
and  in  some  cases,  extremely  difficult  to  allocate  resources  to  learning  events 
in  the  way  they  wanted  to  make  the  assignments.  The  main  problem  encountered 
was  in  the  extremely  limited  number  of  different  resources  allowed  on  MODIA 
(only  30).  All  interface  team  members  had  to  "package"  resources  in  order  to 
make  resource  assignments  using  the  UI.  In  some  cases,  a considerable  number 
of  resources  had  to  be  lumped  together  or  packaged  in  order  to  make  the 
resource  assignment.  In  the  case  of  planning  both  the  30434  and  the  32831 
baseline  courses,  very  few  of  the  needed  resources  could  be  assigned  in  such 
a way  as  to  depict  realistic  use  of  resources. 
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d.  The  limited  capability  of  MODIA  to  handle  the  required  resource 

assignment  was  commented  on  frequently.  In  fact,  the  interface  team  members 

i 

felt  that  the  30  resource  limitation  hampered  realistic  simulation  of  course 
operation  since  the  32831  basic  course  required  the  assignment  of  80  different 
resources  in  the  30434  baseline  course,  over  100  different  resource  assignments 
were  needed. 

e.  Another  major  constraint  of  the  UI  program  was  its  inability  to  handle 
certain  student  arrival  options.  For  example,  staggered  entries  with  variable 
numbers  of  entering  students  could  not  be  simulated.  Moreover,  all  interface 
team  members  voiced  the  need  for  a system  which  could  realistically  depict 
shift  operations.  While  the  MODIA  system  can  be  manipulated  to  allow  simula- 
tion of  courses  with  a shift  operation,  the  resultant  product  had  severe 
limitations.  Specifically,  the  250  learning  event  limit  for  the  UI  was  much 
too  restrictive  for  depicting  courses  with  shift  operations.  For  example. 
Course  3ABR32831  would  require  approximately  400  learning  events  to  simulate 
shift  operation.  Resources  and  learning  events  for  this  course  would  have  had 
to  be  condensed  or  packaged  even  more  to  simulate  a two  shift  operation.  More- 
over, course  managers  felt  they  had  a better  handle  on  managing  shift  opera- 
tions with  current  techniques. 

f.  All  interface  team  members  felt  confident  in  using  the  UI  and  all  found 
the  User's  Guide  (provided  by  Rand)  very  helpful  in  working  the  system.  But 
again,  there  was  considerable  difficulty  in  working  around  garbling  problems. 

As  in  phase  I,  the  numbers  143  and  168  were  read  as  145  and  170.  This  particu- 
lar  garb fi;v^  caused  a problem  every  time  one  had  to  enter  learning  event 
numbers  143  or  168.  The  problem  was  surmounted  by  labeling  these  learning 
events  as  "sick-call’1  and  assigning  zero  time  to  the  learning  event. 
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g.  In  general,  the  interface  team  felt  confident  with  the  simulations  and 


expressed  the  need  for  a system  like  MODIA,  but  all  members  also  remarked 
that  the  system,  in  its  present  form,  had  too  many  limitations.  All  inter- 
face members  rated  the  output  of  the  RUM  as  fairly  easy  to  interpret  and  use, 
and  of  value  in  the  planning  process. 

h.  The  specific  changes  recomnended  for  MODIA  are  dealt  with  in  detail  in 
paragraph  15.  Suffice  it  to  say  that  in  the  experience  of^the  Phase  II 
servi/fe  test,  the  UI  system  was  too  large  and  inefficient  to  be  used  on  the 
Keesler  H-6060  computer  now,  or  in  the  future.  The  direction  and  magnitude 
of  the  recommended  changes  to  the  MODIA  system  make  the  specifications  for 
the  present  system  useless  in  supporting  a DAR.  Based  on  limitations  and 
problems  experienced  in  this  portion  of  the  service  test,  and  on  the  results 
reported  in  the  next  section,  MODIA  should  not  be  adopted  for  use  "as  is". 

The  operating  limitations  experienced  during  Phase  II  were  very  similar  to 
those  experienced  in  Phase  I. 

1 2 . MODIA  Usefulness  as  a Planning  Tool. 

a.  General.  The  basic  thrust  of  the  Phase  II  evaluation  effort  was 


to  determine  MODIA' s usefulness  as  a management  planning  and  problem-solving 
tool.  The  assessment  of  MODIA's  utility  must,  of  necessity,  be  subjective 
and  depend  on  the  opinions  and  judgments  of  those  in  the  training  center 
managment  hierarchy  who  would  use  a system  such  as  MODIA.  The  strategy  of 
this  evaluation  effort  was  to  present  course  managers  with  MODIA  products 
(the  RUM  simulation  and  Cost  Model  course  costs  for  each  alternative  design), 
and  see  if  they  could  use  either  the  simulation  information  or  cost  model 
information  to  arrive  at  more  cost  effective  course  designs  --  while  staying 
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within  the  limits  of  ATC  and  Keesler  TTC  policy  directives,  manpower 
limitations  and  resource  constraints. 

The  basic  question  involves  "how"  management  should  use  the  hODIA 
system.  Therefore,  management  responses  and  results  of  alternative  course 
costs  will  be  discussed  with  respect  to  obtained  results  in  each  of  the 
technical  training  groups,  respectively.  To  help  the  reader  keep  this  part 
of  the  evaluation  conceptually  straight,  there  are  two  basic  aspects  of 
MODIA  information  that  could  be  of  use.  The  questions  that  address  these 
aspects  are:  (1)  How  useful  was  the  cost  model  information  on  alternative 
course  designs?  and,  (2)  How  useful  was  the  simulation  information?  The 
first  question  is  answered  by  results  discussed  here,  while  results  pertain- 
ing to  the  second  question  are  discussed  in  para  13.  In  the  discussion  to 
follow,  each  of  five  basic  courses  in  the  technical  training  groups  was 
simulated  and  cost  for  the  courses  calculated  using  the  cost  model.  The 
"baseline"  course  cost  figures  and  operating  parameters  were  designed  to 
reflect  the  way  the  course  actually  operated  during  1977.  All  baseline 
course  costs  were  figured  on  the  6-hour  training  day  and  were  compared  with 
total  course  cost  figures  derived  using  the  ATC  Comptroller's  figures  on 
costs/graduate  in  each  of  the  training  courses  multiplied  by  the  annual 
graduates.  The  results  show  that  the  cost  model  figure  for  total  course 
costs  agrees  closely  with  a total  course  cost  using  the  comptroller's  cost 
factors  — a result  that  agrees  with  Phase  I findings  on  cost  model  accuracy. 

Because  of  the  time  limits  of  the  service  test,  cost  model  information 
was  obtained  for  three  basic  courses  and  alternatives  for  each  of  the  three 
courses. 
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b . 3380  TTG  - Results  of  Alternative  Costing : 


Table  1 presents  the  cost  of  the  baseline  3ABR32831  course  in 
comparison  with  costs  of  various  alternatives  generated  by  the  interface 
team  members,  ISO  people,  and  training  managers. 

While  the  cost  model  was  designed  to  be  used  either  in  conjunction 
with  the  simulation  portion  or  by  itself,  ideally,  planners  would  cost  out 
alternative  courses  they  had  simulated  to  determine  the  most  cost  effective 
option.  In  the  present  case,  five  out  of  eight  alternatives  were  simulated. 
Not  all  could  be  simulated  due  to  the  time  limitations  of  the  service  test 
and  the  operating  hours. 


COST  MODEL  BASELINE  COURSE  COST  COMPARISONS 
WITH  ALTERNATIVE  COURSE  COSTS 
(3ABR32831 )a 


(Factors  provided  by  Hq  ATC/Management  Analysis) 
6-hr  day  (Baseline) 

8-hr  day 


Alternatives  7 and  8 were  less  expensive  than  the  other  alternatives. 
These  two  courses  represented  slightly  increased  course  hours  over  the  base- 
line course  (the  course  as  it  actually  operated  on  a 6-hr  day  and  for  527 
course  hours).  The  modest  reductions  of  $80. 3K  (4.3%)  for  alternative  7 and 
$173. OK  (9.21%)  for  alternative  8 were  obtained  by  entering  students  every 
36  hours  instead  of  every  40  hours  as  in  the  baseline,  and  by  decreasing 
the  actual  numbers  of  students  entering  the  course.  Additionally,  students 
were  washed  out  on  an  average  of  217  hours  in  alternative  8,  as  opposed  to 
an  average  of  240  hours  to  washout  in  the  baseline.  The  reduction  in  alternativi 
7 was  due  largely  to  reduced  student  pay,  reduced  student  PCS  costs  and 
instruccor  PCS  costs. 

All  training  managers  felt  the  cost  information  provided  by  the  cost 
model  was  of  very  little  use  to  them.  All  sampled  managers  commented  that  the 
cost  model  information  could  be  used  by  Hq  ATC  level  people  involved  in  making 
decisions  about  policy  impact  on  course  costs.  The  managers  in  this  group 
indicated  that  the  cost  savings  shown  by  using  the  cost  model  related  to  costs 
not  managed  by  center  level  managers  (student  instructor  pay  and  PCS  costs). 
These  obtained  cost  savings  were  in  areas  most  directly  controlled  by  Hq  ATC 
management  actions. 

It  is  significant  that  the  two  money  saving  alternatives  (both  7 and  8) 

* ♦ 

were  generated  independently  of  any  simulation  --  i.e.,  neither  of  those  two 
alternatives  were  put  on  the  RUM.  This  fact  demonstrates  that  the  cost  model 
may  be  used  as  a stand-alone  system,  but  the  overriding  question  as  to  "who" 
should  use  it  is  addressed  later  in  this  report.  Because  of  the  recommenda- 
tions generated  by  training  managers  in  this  group  as  well  as  others,  the 
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the  cost  model  data  was  given  to  HQ  ATC  Comptroller  personnel  in  the 
management  analysis  section  for  further  study  and  comment. 

c.  339b  TTG  - Results  of  Alternative  Costing.  Table  2 presents  the 
cost  of  the  alternatives  run  for  the  3ABR27630-000  course.  In  this  case, 
the  baseline  course  cost  generated  by  the  cost  model  was  within  7%  of  the 
ATC  Comptroller's  figures  for  the  cost  of  the  basic  course  in  1977.  The 
second  alternative  simply  represents  the  course  cost  based  on  an  eight-hour 
training  day  as  opposed  to  a 6-hour  training  day.  The  resultant  savings 
are  trivial  ($4.2K).  The  remaining  alternatives  represent  various  ways  of 
figuring  course  length  based  on  how  policy  dictated  reductions  are  calculated. 
While  the  cost  model  may  give  the  cheapest  alternative  (alternative  2 in  this 
case),  planners  still  needed  to  exercise  judgment.  The  vast  bulk  of  the 
savings  generated  by  this  alternative  was  in  pay  and  allowances  of  students, 
instructors,  and  base  permanent  party  (support)  personnel.  These  costs, 
while  important,  are  not  meaningfully  controlled  by  managers  at  the  training 
group  level.  Additionally,  the  cheapest  alternative  is  not  always  the  best. 

In  the  base  of  3ABR2763U-000,  the  Complementary  Technical  Training  (CTT)  is 
vitally  important  to  the  course  of  training.  Alternative  4 presented  the 
planners  with  the  best  course  length  and  number  of  graduates  from  the  stand- 
point of  meeting  training  standards.  It  is  important  to  note  that  the 
planners  and  managers  already  knew  that  the  mix  represented  in  alternative 
4 would  be  the  best  option  for  them  to  plan  under  the  new  policies.  It  is 
also  important  to  emphasize  that  the  way  in  which  these  policy  decisions 
would  be  implemented  was  determined  independently  of  cost  model  information. 
The  cost  model  data  confirmed  what  managers  already  knew  about  the  effect  of 
recent  policy  decisions  on  course  costs. 
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TABLE  2 

COST  MODEL  BASELINE  COURSE  COST  COMPARISONS  WITH 
ALTERNATIVE  COURSE  COSTS  (3ABR27630-000)a 


Alternative 

Data 

Baseline 

c i d 

2 e 

3 f 

49 

1. 

Annual  Entry 

550 

550 

550 

550 

550 

2. 

Annual  Graduates 

471 

471 

490 

471 

481 

3. 

Annual  Failures 

11 

11 

11 

11 

11 

4. 

Avg  Course  Hours 

215 

285 

215 

274 

242 

5. 

Number  of  Instructors 

25 

25 

25 

25 

25 

6. 

Total  Course  Cost 
(1977  dollars)  b 

1717.1 

1713.9 

1567.4 

1687.1 

1622.7 

7. 

ATC  Course  Cost/Grad 

3.92178 

8. 

ATC  Total  Course  Cost 

1847.2 

9.  Difference  (between  8 and  6)  -130.1  (7%) 


a & b - Same  as  other  tables 

c.  6-hr  day 

d.  8-hr  day 

e.  8-hr  day  (not  adding  CTT) 

f.  8-hr  day  course  + (CTT  - 15%) 

g.  8-hr  day  (Course  + CTT)  - 15% 
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Both  managers  in  this  group  rated  the  cost  model  information  of 
no  use  at  all  to  them.  They  indicated  that  the  information  could  be  of 
use  to  Hq  ATC  Training  Managers  and  others  in  evaluating  the  cost  of  current 
course  training  and  in  evaluating  the  cost  of  alternative  course  designs, 
d.  3410  TTG  - Results  of  Alternative  Costing. 

Table  3 contains  the  results  of  alternative  costing  for  the  3ABR 
30434  course.  Only  two  alternatives  were  run  through  the  cost  model  in  this 
course  because  time  ran  out  for  the  service  test.  A problem  with  the  amount 
of  time  necessary  to  gather  data  for  input  into  MODCOM  (cost  model)  bears 
examination  at  this  point.  This  problem  occurred  with  gathering  data  to 
input  into  all  three  baseline  courses,  but  is  discussed  here  for  convenience. 
All  interface  team  members  spent  a great  deal  of  time  gathering  cost  informa- 
tion and  putting  it  in  a form  usable  by  the  cost  model.  Specifically,  for 
the  3ABR30434  course,  30 % of  the  total  time  spent  by  the  interface  team 
members  for  all  phases  of  the  service  test  was  in  gathering  cost  data  for 
MODCOM  (for  interface  team  members  on  3ABR32831  - 50%  and  3ABR27630  - 50%). 
This  amount  of  time  is  grossly  disproportional  when  one  considers  the  in- 
ability of  managers  to  use  the  final  cost  information.  In  any  event,  the 
interface  team  felt  that  entirely  too  much  time  was  spent  on  this  portion 
of  the  service  test.  The  mechanics  of  inputting  the  information  and  obtain- 
ing final  products  was,  on  the  other  hand,  extremely  easy  and  presented  no 
problems  whatsoever  in  terms  of  computer  usage.  Once  the  baseline  cost 
information  was  obtained,  it  was  very  easy  to  generate  costs  for  alternative 
course  designs. 
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TABLE  3 


COST  MODEL  BASELINE  COURSE  COSTS  COMPARISONS  WITH 
ALTERNATIVE  COURSE  COSTS  (3ABR30434)  a 


Alternative  # 

Basel ine 

1 c 

2d 

Data 

1. 

Annual  Entry 

591 

591 

591 

2. 

Annual  Graduates 

402 

424 

435 

3. 

Annual  Failures 

25 

26 

26 

4. 

Avg  Course  Hours 

460 

520 

460 

5. 

Number  of  Instructors 

73 

73 

73 

6. 

Total  Course  Cost 
(1977  dollars)  b 

3280.2 

3105.1 

2986.9 

7. 

ATC  Course  Cost/Graduate 

8.46088 

8. 

ATC  Total  Course  Cost 
((7)  X (2)) 

3401 . 2 

9. 

Di f ference 

(Between  8 and  6)  - 121.00  (-3.5%) 


a.  In  thousands  of  dollars 

b.  Adjusted  by  a factor  of  1.1598  X MODCOM  value  X 1.057  - 1977  dollars 
(Provided  by  Hq  ATC/Management  Analysis) 

c.  8-hr  day  - same  course  length 

d.  (Course  + CTT)  - 15% 
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The  two  alternatives  presented  for  the  30434  course  show  a 


roughly  5%  saving  (alternative  1)  for  the  course  planned  on  a straight 
8 hour  training  day  with  no  CTT  added;  and  a roughly  9%  saving  when  plan- 
ning a course  length  with  CTT  added  and  reducing  the  resultant  course  hours 
by  15%.  As  with  the  other  training  groups,  this  information  was  not  enlight- 
ening to  training  managers.  One  of  the  managers  found  the  cost  model  infor- 
mation very  useful  in  improving  course  manning  structure  and  student  use  of 
equipment.  Both  managers  felt  that  costs  were  generally  determined  by  ATC 
policy,  rather  than  managed  by  training  group-level  personnel. 

13.  MODIA's  Usefulness  as  a Problem-Solving  Tool  — The  Resource  Utiliza- 
tion Model  Simulation.  Based  on  the  responses  to  questionnaires  (see  Appendix 
5)  given  to  the  training  managers  in  the  three  training  groups  involved  in  the 
Phase  II  service  test  and  on  interviews  with  group  and  center-level  manage- 
ment, the  RUM  does  not  appear  to  be  a useful  planning  tool  that  can  be  used 
by  these  managers  to  more  effectively  manage  training  resources. 

a.  General : As  mentioned  in  the  Methodology  section,  all  managers  were 
presented  with  completed  course  simulations  and  cost  model  data  for  all 
alternatives  --  all  managers  had  the  products  carefully  explained  to  them 
(8/9  of  the  managers  responded  that  they  had  the  products  explained  well 
enough  to  them  that  they  understood  all  the  products  from  the  RUM  and  the 

* 

cost  model).  All  managers  were  facing  real  course  management  problems  at  the 
time  of  the  evaluation.  For  example,  all  had  to  revise  courses  from  a six 
to  an  eight  hour  training  day,  all  had  instructor  shortages,  and  all  had 
students  awaiting  training.  The  RUM  simulation  was  unable  to  provide  the 
training  managers  with  unique  information  on  course  operation.  From  their 
responses  to  questions  1,  2,  9,  11,  12,  13,  14,  15,  16  and  17,  it  was  apparent 
that  the  MODIA  simulation  was  not  telling  the  managers  something  they  did 
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not  already  know  about  specific  problems  in  the  operation  of  the  courses 
studied  in  this  service  test.  It  appeared  from  responses  given  in  the 
questionnaire  that  the  RUM  simulation  was  generally  of  very  little  use  to 
the  group  level  managers.  From  responses  generated  during  debriefing  and 
from  comments  on  the  questionnaires,  the  problems  concerning  the  managers 
were  foreseen  without  the  aid  of  the  simulation.  The  more  pressing  problems, 
such  as  those  concerned  with  shift  operation,  could  NOT  be  realistically 
simulated  on  MODIA.  Seven  managers  said  they  had  little  confidence  in  the 
RUM  simulation,  and  all  nine  managers  felt  they  had  foreseen  the  problems 
in  course  operation  just  as  well  or  better  than  the  simulation. 

At  this  point  it  is  of  interest  to  note  that  enthusiasm  for  MODIA 
ran  high  during  the  service  test  because  of  perceived  potential  of  the  system 
for  helping  managers  solve  some  of  the  problems  that  were  facing  them  at  the 
time.  However,  the  managers  expressed  frustration  with  the  MODIA  system  when 
they  could  not  use  it  to  help  them  manage  those  problems.  For  example,  one 
branch  chief  would  have  liked  a system  which  would  allow  him  to  strategically 
pull  instructors  to  support  Air  Force  exercises  and  still  optimally  operate 
the  course. 

At  the  conclusion  of  the  service  test,  several  managers  expressed 
the  need  for  a computerized  system  which  would  help  them  with  scheduling 
problems,  and/or  a system  which  would  optimize  the  use  of  certain  resources. 

It  was  explained  to  the  managers  that  MODIA  was  neither  a scheduling  nor  an 
optimizing  tool.  One  has  to  use  the  RUM  simulation  to  test  the  feasibility 
of  the  given  design  the  planner  brings  to  the  system.  Other  systems,  such 
as  the  Advanced  Instructional  System  (AIS),  could  be  used  to  resolve  the 
scheduling  and  optimizing  problems  which  seem  to  represent  the  more  important 
management  problems  facing  course  managers. 


L 
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The  results  of  the  managers'  opinions  about  the  simulation  differed 
slightly  from  the  opinions  of  the  ISO  participants  and  from  the  judgments 
of  the  interface  team  members.  Two-thirds  of  the  ISD  people  felt  the  RUM 
simulation  would  be  of  little  use  in  course  planning,  while  two-thirds  of 
the  Interface  team  members  felt  it  would  be  useful.  Both  ISD  respondents 
and  Interface  team  members  were  confident  in  the  simulations  of  their 
respective  courses.  Of  the  six  respondents  (3  ISD  people  and  3 interface 
team  members),  three  said  they  would  seldom  use  MODIA  were  it  to  become  a 
fully  operational  system,  and  three  said  they  would  use  the  system  often. 

This  result  compares  to  the  opinions  of  the  training  managers,  where  five 
of  nine  said  they  would  at  least  use  the  system  "sometime"  if  it  were  fully 
operational . 

In  summary,  the  opinions  of  those  involved  with  the  Phase  II  service 
test  found  the  simulation  lacking  in  certain  respects.  In  general,  there 
were  mixed  feelings  about  the  usefulness  of  the  RUM  simulation.  The  managers 
felt  that  the  simulation  was  of  little  value  to  them,  but  the  ISD  and  inter- 
face team  members  were  of  the  mixed  opinion  that  perhaps  there  was  some  value 
to  be  had  in  the  way  MODIA  simulated  course  operation.  All  individuals  sampled 
with  the  questionnaire  felt  that  the  simulation  of  resource  use  was  less  than 
totally  realistic  and  those  most  closely  involved  with  MODIA  expressed  serious 
reservations  about  the  restrictive  limits  on  the  number  of  resources  that 
could  be  planned  using  MODIA.  A summary  of  the  training  managers'  responses 
in  each  training  group  is  presented  below. 

b.  The  3380  TTG  - 3ABR32831 . The  interface  and  ISD  participants  for 
this  training  group  had  some  difficulty  packaging  resources  for  this  long 
duration  and  high-flow  course.  For  example,  there  were  80  important  resources 
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that  could  not  be  broken  out  as  desired  in  the  basic  course  simulation. 

In  fact,  the  course  structure  as  it  existed  in  actual  operation  could  not 
be  accurately  depicted.  3ABR32831  could  not  be  depicted  as  progressing 
from  group  lock-step,  to  a self-paced  portion,  then  back  to  a group  lock- 
step  again.  The  specific  problem  facing  the  simulation  of  3ABR32831  was 
that  the  students  had  to  be  returned  to  that  portion  of  the  self-paced 
block  from  which  they  were  taken  in  order  to  complete  the  last  group-lock- 
step block.  The  configuration  that  was  simulated  had  a self-paced  portion 
at  the  end  of  the  two  lock-step  portions  with  students  arriving  in  random 
intervals.  The  training  managers  who  examined  the  simulation  felt  that 
such  a simulation  was  of  very  little  use  or  no  use  at  all  to  them  — 5/5 
responses  were  in  this  category;  4/5  of  the  managers  in  this  training  group 
felt  the  simulation  and  cost  model  information  were  of  little  value;  and  3/5 
had  little  confidence  in  the  final  simulation. 

c.  The  3390  TTG  - 3ABR27630  Course.  The  two  training  managers  in  the 
3390  Technical  Training  Group  found  MODIA  to  be  more  useful  than  did  the 
managers  in  the  other  two  training  groups.  Both  felt  MODIA  would  be  useful 
in  helping  them  manage  course  revisions  better,  both  thought  the  system 
had  value  to  them  as  managers,  and  both  were  very  confident  in  the  results. 
The  difference  in  the  response  of  this  group  and  the  others  can  be  attributed 
to  the  fact  that  MODIA  simulation  helped  the  managers  spot  a queuing  problem 
that  existed  in  the  27630  course  operation.  While  the  managers  knew  a 
problem  of  some  kind  existed,  it  seems  MODIA  highlighted  a possible  solution 
which  was  subsequently  put  into  operation  --  the  queuing  problem  was  solved. 

While  MODIA  generally  provided  favorable  results  in  this  training 
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group,  several  comments  by  the  managers  are  important  in  assessing  MOOIA's 
usefulness.  First,  MODIA  could  not  adequately  simulate  the  group-paced 
operation  of  the  course.  MODIA,  however,  can  be  manipulated  to  handle  group- 
paced  instruction,  but  in  the  present  case  the  options  available  for  simulating 
group-paced  instruction  were  not  acceptable  to  training  managers.  The  managers 
were  not  satisfied  with  the  way  the  resultant  course  "looked"  in  the  simulation. 
Second,  MODIA  didn't  allow  the  managers  to  more  effectively  manage  resources. 

As  the  branch  chief  remarked,  "In  its  present  form,  the  only  useful  purpose 
it  serves  is  to  highlight  the  facility  costs  in  one  single  document."  The 
most  pressing  problems  facing  these  managers  were  instructor  manning  shortages. 
They  felt  it  would  be  futile  to  exercise  a system  that  merely  highlighted 
the  manning  problems  they  were  aware  of  already. 

d.  The  3410  TTG  - 3ABR30434.  The  most  predominant  remarks  made  by 
managers  In  this  group  dealt  with  the  limitations  of  the  MODIA  system. 

Both  managers  felt  that  MODIA  in  its  present  form  would  be  of  little  value 
and  they  had  little  confidence  in  the  simulation.  They  both  said  that  they 
would  use  MODIA  often,  if  it  were  substantially  changed. 

14.  Organizational  Configuration,  Operational  Procedures,  and  Resources 
Required  to  Implement  MODIA. 

As  stated  in  the  MODIA  Evaluation  Plan  for  the  Phase  II  service  test, 
the  determination  of  how  MODIA  should  be  used  largely  depended  on  how  well 
the  planners  and  managers  felt  they  could  use  the  simulation  and  cost  informa- 
tion. The  results  of  the  two  preceding  sections  indicate  that  the  RUM  simula- 
tion of  course  operation  was  of  too  limited  a scope  to  be  of  any  value  in 
the  planning  and  management  of  course  operation  at  the  center,  group  or 
branch  levels. 
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All  personnel  involved  in  the  service  test  were  queried  as  to  how 
and  where  MOD I A should  be  used  if  it  were  adopted.  There  was  a wide  range 
in  the  recommendations  as  to  who  should  use  MODIA.  Some  managers  felt  that 
only  branch  level  planners  should  use  the  system,  while  most  others  recom- 
mended use  by  everyone  involved  in  the  planning  process,  from  branch  level 
to  Hq  ATC  course  managers  and  manpower  personnel.  Many  managers  stated  they 
would  not  recommend  the  system  as  it  exists  now,  but  stated  that  they  could 
use  a MODIA-like  system.  A particular  surprise  was  the  suggestion  by  many 
that  Hq  ATC  level  personnel  could  use  cost  model  information.  This  recom- 
mendation was  surprising  in  light  of  the  fact  that  at  the  beginning  of  the 
service  test,  managers  and  other  training  center  personnel  expressed  a fear 
that  MODIA  would  be  used  by  headquarters  to  impose  unrealistic  course  policy 
changes  on  them.  That  training  managers  thought  enough  of  the  cost  model  to 
recommend  its  use  by  Hq  ATC  planners  speaks  well  for  the  cost  model.  Again, 
though,  almost  all  the  personnel  questioned  did  emphasize  that  the  simulation 
should  not  be  used  by  anybody  unless  it  was  considerably  changed. 

The  comments  and  responses  about  the  best  organizational  configuration 
were  clear  in  the  recommendations  that  one  centrally  located  interface  team 
could  handle  the  planning  of  all  Type  3 courses  at  Keesler.  By  far  the 
most  frequent  recommendation  was  that  a well -trained  interface  team  composed 
of  only  3 members  could  handle  all  the  necessary  planning. 

In  addressing  the  questions  of  resources  required  to  implement  MODIA, 
the  results  of  this  service  test  have  several  clear  implications.  As  far 
as  the  manpower  required  to  operate  the  system,  the  results  of  this  evaluation 
indicate  that  very  little  manpower  increases  would  be  needed  to  operate  the 
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system  effectively.  This  result  is  consistent  with  the  most  prevalent 
recommendation  in  this  service  test  --  that  MODIA  be  operated  by  a 
centrally  located  team  of  about  3 individuals.  The  unexpectedly  low  cost 
of  this  service  test  was  achieved  for  a variety  of  reasons,  dealt  with 
fully  in  the  discussion  section,  but  in  general  the  results  indicate  much 
fewer  manhours  involved  in  operating  the  system  than  may  have  been  estimated 
based  on  Phase  I results. 

In  contrast  to  the  small  manpower  increases  that  would  be  required  to 
implement  MODIA,  the  results  of  the  service  test  indicate  considerable 
expenditures  in  computer  resources  would  be  required  to  implement  the  MODIA 
system. 

The  severe  interference  with  training  caused  by  operating  the  UI  and 
the  resultant  restriction  in  operating  hours  for  the  service  test  indicate 
that  MODIA  as  it  currently  is  written  could  NOT  be  used  for  eight  hours  a 
day  without  causing  unacceptable  impairment  of  other  training  being  conducted 
on  the  H-6060.  There  appears  to  be  little  possibility  of  using  MODIA  on 
the  B-3500  system,  since  Hq  ATC/ACD  has  gone  on  record  as  stating  the  B-3500 
system  is  currently  saturated.  The  computer  personnel  at  Keesler  felt  MODIA 
could  not  under  any  circumstances  be  used  in  its  present  form,  since  exist- 
ing computer  resources  and  current  training  priorities  leave  little  room 
for  a system  as  large  as  MODIA.  The  unacceptability  of  MODIA  "as  it  currently 
exists"  is  a consistent  theme  that  runs  through  the  comments  of  all  those 
involved  with  the  service  test.  The  recommendations  advanced  for  making 
MODIA  more  acceptable  and  usable  are  discussed  in  the  next  section. 

15.  Necessary  Changes  to  Improve  MODIA  Effectiveness.  One  of  the  standout 
results  of  this  evaluation  was  that  MODIA  would  have  to  be  dramatically 
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changed  if  it  were  to  be  an  effective  planning  tool.  Far  and  away  the 
most  prevalent  recommendation  for  change  in  the  RUM  was  that  the  limit  on 
the  number  of  training  resources  be  considerably  increased.  The  current 
limit  of  30  resources  is  just  not  adequate.  All  courses  planned  in  this 
service  test,  as  well  as  one  of  the  courses  in  the  Phase  I evaluation,  had 
difficulty  working  around  this  limitation.  The  magnitude  of  the  problem 
created  by  a limit  of  30  resources  is  highlighted  when  one  considers  that 
the  average  number  of  resources  used  in  most  courses  is  considerably  larger 
than  30  (and  can  go  as  high  as  1100  resources  in  one  particular  course). 

a.  In  addition  to  increasing  the  resource  limitations  on  the  UI  and  RUM 
portions  of  MODIA,  it  is  necessary  to  decrease  the  overall  size  of  the  UI 
program,  especially  the  "C"  phase  of  the  UI.  This  phase  requires  70k  bytes 
of  storage  in  a time-sharing  system  with  11  OK  bytes  available  for  users. 

The  large  portion  of  the  time-sharing  system  required  by  this  phase  causes 
unacceptable  interference  with  other  users  of  the  time-sharing  system. 

This  particular  recommendation  for  reduction  in  the  size  of  the  UI  is  a 
result  which  was  also  obtained  in  the  Phase  I service  test. 

b.  At  this  point  it  must  be  mentioned  that  while  the  UI  could  be  rewritten 
to  be  more  efficient  and  still  handle  the  recommended  size  increases  dis- 
cussed below,  the  resultant  size  increases  in  the  operation  of  the  RUM  would 
probably  prove  unacceptable.  Since  the  limits  that  apply  to  the  UI  directly 
affect  the  amount  of  storage  required  by  the  RUM,  increases  in  the  limits 
allowed  on  the  UI  would  greatly  increase  the  amount  of  computer  time  and 

core  storage  demanded  by  the  RUM. 

c.  In  relation  to  the  problem  of  the  overall  size  of  the  UI  program 
(especially  the  1 R ' and  ' C * phases),  the  interface  team  recommended  that 


-29- 


provision  be  made  in  the  program  to  enter  a given  phase  at  particular 
points  during  the  phase.  As  the  programs  are  currently  written,  the  user 
must  enter  a phase  at  the  beginning  if  he/she  is  to  make  a change  and  the 
user  must  go  through  the  entire  phase  and  reenter  all  subsequent  phases. 

As  a consequence,  a considerable  amount  of  time  can  be  expended  for  relatively 
minor  changes  in  the  resource  assignments  or  capacities.  Aside  from  having 
to  reenter  all  subsequent  information  and  the  amount  of  time  and  effort 
involved,  having  to  tie  up  the  computer  for  relatively  minor  changes  impairs 
the  cost  effectiveness  of  the  UI. 

d.  While  shift  operations  can  be  planned  on  MODIA  as  it  currently 
exists,  the  programs  should  be  rewritten  so  as  to  allow  more  di rect  simulation 
of  the  shift  operation.  Current  options  on  MODIA  necessitate  manipulation 

of  the  MODIA  system  in  such  a way  as  to  make  simulation  of  shift  operations 
unrealistic  and  unacceptable  to  training  managers. 

e.  Apparent  from  the  garbling  of  certain  numbers  and  letter  combinations 
and  the  large  size  of  MODIA  programs  was  a certain  amount  of  incompatibility 
between  the  MODIA  software  and  the  Honeywell  system.  MODIA  programs  currently 
cannot  be  rewritten  to  alleviate  this  basic  incompatibility,  but  should  MODIA 
eventually  be  adopted,  the  garbling  problem  would  have  to  be  resolved. 
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SECTION  IV  - DISCUSSION 


16.  General . The  Phase  II  evaluation  effort  differed  from  Phase  I in 
that  MODIA  planning  was  attempted  in  an  operating  environment  with  planners 
and  managers  judging  and  considering  MODIA  simulation  and  cost  information 
in  the  light  of  current  policy  guidelines  and  resource  constraints.  The 
results  of  their  judgments  and  the  operating  experiences  of  this  service 
test  indicate  that  the  simulation  has  very  limited  value  for  the  management 
of  technical  training  courses  at  the  branch,  group,  and  center  level.  This 
result  is  at  apparent  odds  with  Phase  I findings  which  seem  to  indicate  MODIA 
had  potential  for  helping  management  design  more  cost-effective  courses. 

In  the  Phase  I effort,  however,  the  primary  concern  was  determining 
whether  the  simulations  of  course  operations  were  "valid",  i.e.,  could  they 
realistically  simulate  the  way  courses  actually  operated;  and  whether  the 
cost  of  MODIA  could  be  offset  by  more  cost-effective  designs  --  as  compared 
with  conventionally  designed  courses.  Phase  II  mainly  tried  to  determine  how 
course  managers  might  use  the  system.  The  results  clearly  indicate  that 
the  simulation  and  cost  information  was  of  little  use  to  managers. 

No  attempt  was  made  to  compare  MODIA  planning  with  conventional  planning. 

Such  comparisons  are  a little  like  comparing  apples  and  oranges.  Were  MODIA 
to  be  radically  changed  (as  indicated  in  Section  III,  para  15),  the  interface  tea 
and  2 of  3 ISD  planners  felt  the  system  could  be  an  important  aid  in  organizing 
and  clarifying  the  planning  process.  In  short,  MODIA  could  have  been  an 
important  addition  to  the  planning  process,  but  limitations  in  the  system 
as  it  currently  exists  precluded  managers  from  seeing  it  as  a positive,  useful 
tool.  Specific  highlights  of  the  results  will  be  discussed  in  relation  to 
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Phase  I results  and  in  relation  to  MODlA's  potential  for  implementation 
now  and  in  the  future. 

17.  MODIA  Costs.  One  of  the  most  striking  differences  between  Phase  1 and 
Phase  II  service  tests  was  the  costs  of  implementing  and  operating  the  MODIA 
system.  The  main  difference  in  the  cost  of  the  two  service  tests  was  in 
the  reduced  manpower  required  to  operate  MODIA.  This  reduction  reflects 
the  high  level  of  competence  achieved  by  the  interface  team.  The  experiences 
of  Phase  1 seemed  to  have  paid  some  dividends  for  the  Phase  II  effort. 
Particular  mention  must  be  made  of  the  quality  of  training  of  the  interface 
team.  The  interface  team  members,  as  well  as  the  project  officer,  were  very 
adept  at  working  around  problems  and  seemed  to  know  a great  deal  more  about 
how  to  use  the  system  than  may  have  been  the  case  in  Phase  I.  All  four 
individuals  expressed  complete  satisfaction  with  the  thoroughness  of  the 
training  received  at  Rand  Corporation  in  Santa  Monica.  The  monies  expended 
for  this  training  undoubtedly  paid  big  dividends  both  in  the  reduced  cost 
of  operating  the  system  and  in  the  quality  of  the  generated  products. 

Just  how  much  this  experience  of  the  service  test  would  affect  the 
per  hour  cost  of  MODIA  planning  figured  in  the  Phase  I effort  is  difficult 
to  guess  exactly  since  this  service  test  did  not  address  the  cost  of  MODIA 
planning  vis-a-vis  conventional  planning.  It  is  safe  to  say  MODIA  planning 
would  not  be  as  expensive  as  indicated  in  the  Phase  I test.  The  carefully 
kept  work  logs  and  the  comments  by  virtually  all  participants  (Interface 
Team,  ISD,  and  Training  Managers)  indicated  MODIA  planning  would  not  unduly 
complicate  the  planning  process.  They  felt  the  simulation  would  be  a useful 
aid  in  course  planning  were  its  limitations  corrected  to  allow  more  realistic 


representation  of  resource  use  and  more  realistic  simulation  of  course 
shift  operations.  The  limitation  of  the  UI  and  the  RUM  products  appears 
to  be  the  main  factor  mitigating  against  MODIA  effectiveness. 


The  one  "cost"  of  MODIA  that  remained  consistent  across  both  service 
tests  was  the  demand  placed  by  MODIA  on  computer  resources.  These  costs 
were  large  in  both  service  tests.  The  size  of  the  MODIA  programs  was  the 
major  factor  contributing  to  demands  on  computer  resources.  This  result 
was  expected  since  MODIA  programs  were  virtually  unchanged  since  Phase  I 
results  were  published.  It  was  exceedingly  difficult  to  operate  the  system 
without  interfering  with  other  training  being  conducted  on  the  Honeywell 
H-6060.  The  restrictive  operating  hours  that  had  to  be  imposed  as  a result 
of  the  Interference  no  doubt  hindered  the  ability  of  the  interface  team 
to  explore  as  many  alternative  course  simulations  as  would  have  been  desirable 
for  this  service  test.  In  fact,  this  evaluator  was  more  than  a little 
surprised  to  see  as  many  alternatives  generated  as  were  finally  planned  on 
the  system.  The  recommended  changes  to  the  MODIA  system  were  very  similar 
to  recommended  changes  made  in  Phase  I. 

18.  MODIA  System  Limitations.  A look  at  the  recommended  changes  to  MODIA 
provides  the  reader  with  a base  from  which  to  judge  the  limitations  of  the 
system.  The  standout  limitation  is  the  restricted  number  of  resources  that 
can  be  assigned  to  learning  events.  This  limitation  probably  reduced  the 
acceptability  of  the  simulation  to  the  majority  of  the  training  manaqers. 
Aqain,  this  findlnq  Is  unenliqhteninq  to  some  extent  since  Phase  I findinqs 
indicated  that  the  limitations  degraded  its  acceptability  to  course  planners. 
Phase  II  results  showed  the  limitations  degraded  the  system  acceptability 
to  managers  as  well. 
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In  addition  to  the  resource  limits,  the  interface  team  reiterated  the 
desire  to  have  a user  interface  that  would  be  more  adaptable.  They  indicated 
a need  for  a system  which  could  be  entered  at  more  points  in  a given  phase, 
and  which  could  accept  changes  within  a phase  without  having  to  reenter  all 
subsequent  information  within  a file. 

Another  major  factor  influencing  acceptability  of  MODIA  was  the  inability 
of  planners  to  realistically  simulate  courses  with  shift  operations,  and 
courses  with  certain  configurations.  3ABR32831  had  certain  portions  of  the 
course  where  students  progressed  from  lock-step  through  self-paced  instruc- 
tion and  then  back  to  lock-step  again.  The  course  could  only  be  simulated 
with  the  self-paced  portion  at  the  end  of  the  two  lock-step  blocks.  While 
this  resolved  the  problem  of  simulating  the  course  with  MODIA,  it  made  the 
resultant  "picture"  of  course  operation  unrealistic.  The  problem  of  simulating 
courses  with  shift  operations  was  more  serious.  Managers  expressed  a definite 
need  for  accurately  simulating  this  type  of  operation,  but  the  MODIA  system 
was  not  designed  to  handle  shift  operations  in  a way  that  would  be  useful  to 
course  managers. 

In  planning  3ABR27630,  the  managers  were  chagrined  by  MODIA' s inability 
to  simulate  group-paced  instruction.  The  course  was  simulated  using  the  lock- 
step  option,  but  again,  the  resultant  simulation  was  somewhat  unrealistic 
and  the  training  managers  expressed  their  dissatisfaction  with  the  resulting 
product. 

In  sum,  while  some  realistic  simulation  of  the  three  courses  was  achieved, 
managers  felt  that  inherent  limitations  of  the  MODIA  system  prevented  the 
simulation  from  being  of  any  value  to  them  in  managing  training  courses. 
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A majority  of  the  managers  liked  the  MODIA  concept,  but  wanted  the  system 
to  do  more  than  it  was  designed  to  do. 

As  mentioned  earlier,  there  are  other  scheduling  and  optimizing  models 
such  as  A1S,  which  better  handle  the  problems  facing  course  managers. 

A question  naturally  arises  about  the  relatively  small  numbers  of 
managers  exposed  to  the  MODIA  simulation  in  this  service  test.  From  a 
rigorous  standpoint  it  would  be  unwise  to  generalize  about  the  value  of 
MODIA  as  a management  tool  based  on  the  judgments  of  only  nine  course 
managers.  These  nine  were  sampled  in  this  service  test  because  they  were 
most  directly  involved  in  the  management  of  the  courses  selected  for  the 
service  test  and  in  the  best  position  to  judge  the  utility  and  accuracy  of 
the  MODIA  simulation  and  cost  model  information.  Additionally,  the  large 
reorganization  of  the  former  SAAS  under  the  training  center  caused  other 
managers  who  would  otherwise  be  involved  to  be  shifted  to  other  organiza- 
taional  positions.  Only  those  managers  who  could  best  judge  the  accuracy 
and  usefulness  were  asked  to  comment  on  the  system. 

Arguing  for  the  general izability  of  the  training  managers'  judgments 
is  the  fact  that  the  perceived  limitations  of  the  system  were,  by  and  large, 
the  same  limitations  uncovered  in  the  Phase  I service  test  by  course  planners. 
The  fact  that  these  limitations  were  also  judged  by  the  training  managers 
as  constraining  the  usefulness  of  the  simulation  to  management,  provides  a 
reasonable  clue  as  to  the  value  of  MODIA  simulation  to  others.  It  is  argued 
here  that  if  the  managers  must  familiar  with  course  operation  could  not  find 
the  simulation  useful  or  acceptable,  no  one  else  could  either.  The  point 
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has  to  be  made  that  any  manager  looking  at  a somewhat  unrealistic  repre- 
sentation of  course  operation  may  run  the  risk  of  taking  management  actions 
based  on  misleading  information. 

19.  The  Cost  Model . In  general,  the  training  management  found  the  costing 
of  course  alternatives  very  interesting  but  of  little  value  to  them  in  manage- 
ment of  course  operation  or  in  planning  revised  courses.  The  large  majority 
of  managers  stated  that  the  cost  model  information  was  of  little  practical 
value  at  their  level,  but  went  so  far  as  to  suggest  the  use  of  cost  model 
information  by  Hq  ATC  level  management.  The  obtained  accuracy  of  the  cost 


model  figures  and  the  ease  with  which  alternative  course  costs  could  be 
generated  argued  strongly  for  its  adoption  at  some  level.  The  inputs  to 
the  cost  model  portion  of  MODIA  can  be  obtained  independently  of  any 
information  provided  by  the  simulation  portion  of  the  MODIA  system.  In 
short,  from  the  results  of  this  evaluation,  MODCOM  could  be  useful  for 
Hq  ATC  level  planners  and  managers.  Based  on  the  service  test  experience, 
however,  certain  caveats  have  to  be  issued  regarding  the  time  involved 
to  gather  and  format  information  for  input  into  the  cost  model. 

a.  The  input  information  for  use  by  MODCOM  took  a long  time  to  gather 
and  put  into  a usable  form.  Regardless  of  who  uses  the  product,  input 
undoubtedly  would  have  to  come  from  branch  and  group  level  planners,  and 
would  have  to  be  updated  regularly  by  the  same  people.  This  effort  would 
naturally  extend  to  all  Type  3 courses  at  each  of  the  training  centers  and 
would  involve  substantial  changes  in  the  way  maintenance  and  cost  data  on 
training  course  resources  are  kept.  Would  the  effort  be  worth  it?  In 
order  to  get  a feel  for  the  utility  of  the  information  provided  by  the  cost 
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model,  the  cost  model  Information  on  each  of  the  three  basic  courses  and 
for  all  alternatives  examined  in  this  service  test  were  given  to  planners 
in  the  Management  Analysis  Directorate  of  the  HQ  ATC  Comptroller.  The 
results  of  their  study  of  MODCOM  indicated  that  the  Cost  Model  would  probably 
not  be  of  ar\y  use  at  the  HQ  ATC  level. 

b.  Despite  the  opinion  of  group  and  branch  level  management  that  MODCOM 
information  was  interesting,  but  of  little  value,  it  could  be  argued  that 
these  managers  should  be  using  this  information  regardless  of  current 
practice.  The  argument  may  go  that  just  because  they  are  not  used  to 
considering  costs  of  alternative  course  designs,  they  could  or  should  make 
such  considerations  when  they  plan  or  revise  courses.  This  position  involves 
the  determi nation  of  "what"  or  "who"  drives  course  costs.  The  managers 
sampled  in  this  exercise  felt  that  training  philosophy,  as  presented  in 
center-level  and  headquarters- level  policy  guidelines  (as  well  as  practical 
considerations  like  the  TPR)  drove  the  bulk  of  the  costs  of  training.  As 
it  turns  out,  current  systems  for  managing  course  costs  are  adequate,  and 
the  cost  model  information  would  not  add  anything  to  current  management  of 
course  costs. 
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SECTION  V - CONCLUSIONS 

20.  MODIA  System  Costs.  The  personnel  costs  associated  with  implementing 
MODIA  at  Keesler  in  the  near  future  would  be  modest.  Three  additional 
positions  would  be  all  that  would  be  needed  to  man  an  effective,  centrally 
located  interface  team. 

Computer  costs  in  terms  of  demands  on  the  time-sharing  system  would 
be  unacceptable.  Current  computer  resources  are  not  sufficient  to  support 
a fully  operational  MODIA  system  as  it  is  currently  structured. 

21.  MODIA  System  Limitations.  The  final  determination  of  the  feasibility 
of  making  the  recommended  changes  to  the  MODIA  System  lies  with  Rand. 

Specific  changes  which,  in  the  judgment  of  the  evaluators,  must  be  made  are: 

a.  Increase  the  limit  of  the  number  of  resources  that  can  be  programmed 
with  the  UI  Program.  An  arbitrary  limit  increase  which  may  prove  acceptable 
would  be  a four  or  five-fold  increase. 

b.  Increase  the  limit  of  learning  events  from  250  to  approximately  400. 

c.  Rewrite  programs  to  be  compatible  with  the  Honeywell  system. 

d.  Rewrite  RUM  and  UI  programs  to  allow  planning  of  group  paced 
courses  and  courses  with  two  shifts. 

e.  Reduce  the  size  of  the  UI  so  as  to  make  the  size  of  various  phases 
much  smaller.  A good  rule  of  thumb  would  be  to  keep  each  phase  within  Air 
Force  standard  of  40k  bytes  of  storage. 

f.  Rewrite  the  UI  so  each  phase  could  be  entered  at  different  points, 
and  subsequent  information  would  not  have  to  be  reentered. 

The  limitations  reflected  in  the  recommended  changes  prevented  the  RUM 
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simulation  of  course  operation  from  being  useful  to  training  managers  at 
the  branch,  group,  and  center  levels. 

The  evaluator  feels  that  the  contradictory  demands  represented  in  the 
recommended  changes,  i.e.,  the  demand  to  drastically  increase  the  size  and 
variety  of  options  offered  in  the  simulation  versus  the  demand  to  decrease 
the  overall  size  and  increase  efficiency  of  the  software,  prevent  the  present 
MODIA  simulation  from  being  a viable  and  cost  effective  planning  tool. 

22.  MODCOM  Usefulness.  After  reviewing  the  MODCOM  data  generated  during 
the  Phase  II  service  test,  the  conclusion  of  the  Management  Analysis 
personnel  was  that  MODCOM  would  not  add  significantly  to  current  cost 
management  techniques. 
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SECTION  VI  - RECOMMENDATION 


23.  The  following  action  is  recommended: 

Recommend  that  the  MODIA  System  not  be  adopted  for 
use  in  ATC  at  this  time. 
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KTTC  COST  INCURRED  TO  IMPLEMENT/OPERATE 
MODIA  PHASE  II  SERVICE  TEST 


CATEGORY 

FACTOR 

RATE 

COST 

I.  IMPLEMENTATION  COSTS* 

A.  Computer  Terminal  Time 

B.  Other  Projects  run  on 

16.53  hr 

$220. 00/hr 

$3737.00 

MODIA  Systems 

15.4  hrs 

220.00/hr 

3391.00 

C.  Corrmunl  cations  line 

installation  costs 
(installation/removal ) 

200.00 

D.  TDY  Costs 

1.  Interface  Team  Train- 

-- 

- - 

3194.00 

ing. 

2.  Lackland  TDY 

— 

— 

262.00 

E.  MODEM  Costs 

5 month 

58.00/Mo 

290.00 

SUB  TOTAL 

11.074.00 

II.  OPERATING  COSTS 


A.  3ABR32831 


I.  Computer  terminal 


time 

44.8  hr 

220.00/hr 

9856.00 

2.  Personnel  usage 

1 E-6  (interface) 

178 

6.45/hr 

184.45 

2 E-5  ( ISD) 

47 

5.39/hr 

253.33 

3 E-5  " 

97 

5.39/hr 

522.00 

4 E-4  " 

22 

4.65/hr 

102.00 

5 GS-9  " 

22.5 

9.43/hr 

212.24 

Sub  Total 

11.130.001 

3ABR30434  (all  three 
courses) 

1.  Computer  Terminal 

16.5  hrs 

220.00/hr 

3,630.00 

time 

2.  Personnel  Usage 

a.  E-7 

133.25 

7.55/hr 

1,006.00 

b.  GS-9 

108.5 

9.43/hr 

1,023.00 

Sub  Total 

5.659.001 

^Rounded  to  nearest  dollar. 

♦Excludes  cost  of  Rand 
personnel  usage  in 
setting  up  MODIA. 
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CATEGORY 

FACTOR 

RATE 

COST 

C.  3ABR27630  (All  three 
courses) 

1.  Computer  terminal 
time 

2.  Personnel  Usage 

a.  GS-11  (Interface) 

b.  GS-9  (ISD) 

50,3  hrs 

33.25 

15.5 

220.00/hr 

11. 41 /hr 
9.43/hr 

$11 ,066.00 

379.38 

146.17 

Sub  Total 

$11,592.00 

OTHER  COSTS 

Personnel  Costs: 

Waiting  (Computer  Proglems  - 
Access,  etc.) 

E-7 

E-6 

GS-11 

7.00 

97.75 

45.25 

7.55 

6.45 

11.41 

$ 52.85 

630.49 
516.30 

SUB  TOTAL 

$ 1,199.00 

TOTALS 

Project  Officer  (Keesler) 

Capt  (69  days  x 8 x .6) 

331.00 

11.01 

$ 3,644.00 

Total  Computer  Terminal  Time  Costs 

$31,680.00 

Total  Personnel  Costs  (Manpower  Time) 

3,828.00 

Total  Implementation 

3,946.00 

Total  Other 

1,199.00 

TOTAL  COST 

$44,297.00 
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STORAGE  REQUIREMENTS1  FOR  VARIOUS  PORTIONS 
OF  THE  USER  INTERFACE 


Appendix  2 
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INSTRUCTIONS  FOR  THE 
INTERFACE  TEAM  QUESTIONNAIRE 

1.  You  have  participated  as  members  of  the  MEDIA  Interface  Team  in 
exercising  the  MQDIA  system  for  planning  technical  training  courses. 

You  are  in  a position  to  make  a number  of  assessments  concerning  MEDIA 
requirements  far  operation,  as  well  as  the  usefulness  of  MEDIA  system 
output  for  you  in  planning  course  revisions.  Below  you  will  find  a 
matter  of  specific  questions  concerning  details  of  your  experience  with 
the  MEDIA  system.  The  judcpients  required  of  you  are  approximate  in 
nature,  nevertheless,  please  exercise  thoughtful  consideration  for  each 
question.  Your  responses  will  be  kept  in  complete  confidence.  Only 
sunmary  statistical  results  of  your  responses,  ocmbined  with  the  responses 
of  others,  will  be  used  in  deciding  on  the  utility  of  the  MEDIA  system 

to  course  planners. 

2.  Please  read  each  question  carefully  and  indicate  your  response  on  the 
rating  scale  by  placing  a check  mark  BETWEEN  one  of  the  five  sets  of 
delimiters.  When  you  are  finished,  make  sure  you  have  completed  the 
general  information  sheet,  and  that  you  have  put  your  name  in  the  indicated 
plac*«  Place  the  oonpleted  questionnaire  in  the  envelope  provided  and  return 

i it  to  TKXJT.  Remember,  no  other  Technical  Training  Wing  personnel  will 
see  your  responses.  The  information  will  be  analyzed  and  presented  in 
summary  statistical  form  by  -the  Technology  Applications  Center  at  Lackland 
AFB,  Texas. 

i.  If  you  have  any  additional  Garments,  a clarification,  or  explanations 
regarding  any  particular  questions,  use  the  back  of  the  sheet  containing 
the  question.  Indicate  the  question  number  by  your  comments. 


How  much  of  the  information  necessary  for  the  input  into  the 
Interface  did  the  ISD  teams  have  ready? 


5.  How  difficult  was  it  to  assign  subject  matter  types  to 
learning  objectives? 


8.  How  easy/difficult  was  it  to  assign  required  resources  to  learning  events' 


• | » ••  *2*  • • •• 

immn cmricuiT lroronicY tast rmiHrcr 

DIFFICULT  DIFFICULT  EASY 


9.  Did  the  User  Interface  provide  you  with  enough  options  for  describing  student 
arrivals? 


; 1 ; 

; 1 . . 

: 1 : 

■Ili.'Will'iV'BiBIB.O] it  IMlAi 

— urn 

ENOUGH 

I WOULD  HAVE 

ENOUGh 

LIKED 

TOT 


THEFT 


I WOULD 
EVER  USE 


10, 


Was  Volume  II  helpful  to  you  in  using  the  User  Interface? 
Yes  = 3 

If  not,  please  briefly  list  the  shortcomings: 


11*  How  helpful  was  the  user  Interface  Guide? 


Wm? VOTTTTTir  — ABEWATE TOTFUI EHBEMETT 

WHATSOEVER  HELP  HELPFUL 


12 • How  many  of  the  planning  factors  the  ISO  teams  considered  important  were  you 
able  to  consider  in  planning  your  course  on  MODIA? 


3 


17.  How  often  did  you  experience  difficulty  operating  the  User  Interface? 


L 


m&r 


: 1 : 


OCCASIONALLY  frequently  alWAYs 


13.  How  often  did  you  have  to  make  special  arrangements  to  use  the  computer? 


: 1 : 


2 • 


occasionally  frequently  Always 


19.  How  easy/difficult  was  it  for  you  to  interpret  the  output  from  the  User 
Interface? 


exyrwely difficult 


DIFFICULT 


ModERatEly  FAsT 
DIFFICULT 


FxtREMely- 


DIFFICULT 


20.  How  easy/difficult  was  it  to  interpret  the  Resource  Utilization  Model 
output  to  the  ISO  teams? 


EXTREMELY 


DIFFICULT 


■MODERATELY 

DIFFICULT 


TXTRFMFCT 

DIFFICULT 


21.  Would  it  unduly  complicate  the  planning  process  if  MODIA  were  used  regularly? 
No  = 3 


If  your  answer  to  the  above  question  was  yes,  please  list  the  reasons  you 
feel  MODIA  would  complicate  the  planning  process: 
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22.  Compared  to  the  conventional  planning  process,  how  much  time  do  you 
think  you  would  save  using  MODIA  to  plan  and  revise  courses? 


SORT 


: : : : 

VEKYTTTTLT  FRHJFRaTe  C0N$iderA3LE 


A GfttAT 
DEAL 


23.  About  how  often  would  you  use  MODIA  in  designing/revising  a course? 


nettr  irasfi  occasionally — tootnit? mm 

"If  redesigned" 

24.  if  all  type  3 courses  at  Keesler  were  planned  using  MODIA,  about  how 
many  interface  team  members  do  you  think  would  be  necessary: 

All  responses  = 2 to  3 


In  your  opinion,  did  the  ISD  teams  use  as  many  of  MODIA' s options  as  they 
could  have? 


25.  How  easy/difficult  a time  did  you  have  simulating  snared  resources 
between  courses? 


: : 1 ; • 1 • • . 

fxTREMELY  diffTCult  I mOdeSAtely Rsy "Eximciv- 

DIFFICULT  DIFFICULT  DIFFICULT 


24.  How  useful  was  the  Resource  Utilization  Model  simulation  of  shared  course 
resources? 


no 

USX 


. 1 • 


LITTLE  MODERATELY  USEFUL 

USE  VZe'TVL 


EXTREMELY 

USEFUL 
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27.  How  much  of  the  training  content  of  your  course  were  you  able  to 
assign  using  the  User  Interface? 


ABOUT  HOST* 

HALF 


29*  List  all  of  the  teaching  methods  you  were  able  to 
using  the  User  Interface? 


I 


3°.  How  many  of  the  required  resources  were  you  able  to  assign  to  subject 
matter  types?  J 


: 2 : 


31.  How  realistic  was  the  assignment  of  resources  (in  your  opinion)  in 
your  course  design?  ¥ '• 


UNREALISTIC 


• 1 • 

• 1 • 


REALISTIC 


REALISTIC 
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32.  How  many  tests  and  reviews  could  you  assign' 


NOT  NEARLY 
AS  MANY  AS 


ADEQUATE 


MORE  THAN 
ENCUGHT 


33.  Did  ths  User  Interface  provide  you  with  enough  options  for  describing 

studant  arrivals? 

: 1 • .2.  • • 


not  Nearly 

AS  MANY  AS 
I WOULD  HAVE 
LIKED 


ADEQUATE 


MORE  THAN 
ENOUGHT 


34.  How  many  increments  did  the  User  Interface  allow  you  to  break  your 
course  into  oonpared  to  conventional  course  revision  and  planning? 


NONE 


VERY 

FEW 


ADEQUATE 


A LITTLE 
MORE  THAN 
NEEDED 


MORE  THAN 
I WOULD 
NORMALLY  USE 


35.  Hct#  much  time  did  it  take  to  design  alternative  course  plans  after 
the  baseline  course  was  put  in  User  Interface? 


A GREAT 

DEAL  CF 
TIME 


MORE  TIME 
THAN  IT  TOOK 
FOR  THE  BASE- 
LINE 


ABOUT  THE 
SAME  TIME 


LESS  THAN 
THE  BASE- 
LINE 


NO  TIME 
AT  ALL 
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36.  About  how  much  of  a deterence  to  designing  alternative  courses  was 
the  time  you  spent  using  the  User  Interface? 


NroncKXft — rswi jnrnm ttcorottoble  a great  - 

AT  ALL  DETERENCE  DETERENCE  DETERENCE  DETERENCE 


37.  How  useful  was  the  User  Interface  program  in  helping  you  consider 
planning  factors  in  revising  and  planning  your  courses? 


HTUSE VERY1  LITTLE ROBERT CONSIDERABLY EYTREmiT'' 

AT  ALL  USE  USEFULNESS  USEFUL  USEFUL 


38.  How  aeny  planning  factors  were  you  able  to  consider  in  designing 
or  revising  a course  using  MODIA? 


• • 

• • 

: 1 : 

: 2 : 

• J 

• • 

• • 

— 11  1 1 li  i|  Mj 

THE  NEEDED 
FACTORS  COULD 
BE  CONSIDERED 

FEW 

NUMBER 

I NEEDED 

39.  List  all  other  problems  or  comments  you  would  like  to  mention  with 
respect  to  the  use  of  the  User  Interface  programs. 


40.  How  useful  was  MODIA  simulation  in  helping  you  spot  problems  in 
your  course  design? 


• • 

• • 


UTKT 

AT  ALL 


T5F  VERY — 
LITTLE  USE 


WrAYELV 

USEFUL 


VERY 

USCFUL 


EXTREMELY 

USEFUL 


41.  How  realistically  were  the  problems  in  course  operation  depicted? 


T07HJ? 

UNREALISTIC 


1 


“SOMEWHAT 

UNREALISTIC 


MODERATELY 

REALISTIC 


2 : 


VERY 

REALISTIC 


VIVIDLY 

REAL 


42.  List  all  the  features  of  the  simulation  which  you  feel  were  unrealistic 
Please  briefly  indicate  why  you  feel  the  particular  feature  of  the  simulatic 
was  not  realistic. 
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43.  How  many  managerial  guidelines  were  you  able  to  Incorporate  into 
your  alternative  course  design? 


• • 


flGRT 


: 2 : 

nm~ 

FEW 


immz 

NUMBERS 


• 1 ■ 

HOST 


'ALT 


44.  How  eesy/difficult  was  It  to  incorporate  these  changes  in  alternative 

course  designs?  * 


• • 


IMPOSSIBLE" 


: 1 : 

iw 

DIFFICULT 


: 2 : 


"east 


txtremeiv 

EASY 


45.  How  easy/difficult  was  it  to  use  MODIA  in  simulating  courses  with 

shared  resources? 


No  responses 

iw^siaiE im1 — 

DIFFICULT 


• • ( , 

Average  m T 

DIFFICULTY 


Extremely 

EASY 


46.  How  valuable  do  you  think  MODIA  simulation  will  be  in  revlsinq/ 
pi Inning  courses  with  shared  resources? 

Not  applicable 


anti 

VALUE 


VERY  LITTLE  MODERATE  VALUABLE' 

VALUE  VALUE 


• • 

imtREir 

VALUABLE 


II.*  t0  conventional  course  revision/planning,  how  much  time  do 

you  think  you  would  save  using  MODIA? 


: 1 : : : 

WR7  R5SKT 

LITTLE  AMOUNT 


• • 

An"  EXTREME 
AMOUNT  OF 
TIME 
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*8.  If  MODIA  wer*  a fully  operational  system,  how  much  do  you  feel  it 
ould  complicate  the  revision/planning  process? 


TORT 


: 3 : 

VERY 

LITTLE 


"TOT 

MUCH 


TxTrTmT 


49.  How  Important  do  you  feel  it  is  to  be  able  to  consider  alternative 

-ours*  designs? 


WKW 

UNIMPORTANT 


wvm fmmiuY 

LITTLE  IMPORTANT 

IMPORTANCE 


: 2 : : 1 : 

w? vrmy- 

IMPORTANT  IMPORTANT 


50.  How  confident  are  you  in  the  results  of  the  simulation  of 
course  operation? 


NOT 

CONFIDENT 
AT  ALL 


WEy 

LITTLE 

CONFIDENCE 


: 1 : : 2 : 

moderately'  TRY 
CONFIDENT  CONFIDENT 


CONFIDENT 


51.  How  often  do  you  think  you  would  use  MODIA  in  the  future  were  it  to 

become  fully  operational? 

% 


; 1 : 


sometime ora 


CONSTANTLY 


52.  How  easy/difficult  to  Interpret  was  the  output  from  the  Resource 
Utilization  Model? 


• • • I • 

UrviMlLV  FEET 

DIFFICULT  DIFFICULT 


. . . 2 . . . 

TOD'ERATELY RST EXTREMET? 

DIFFICULT  EASY 


I 
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53-  How  useful  for  course  planning  was  the  simulation  output  from  the 
Resource  Utilization  Model? 


34.  List  the  planning  factors  that  you  would  normally  consider  in  course 
design  which  you  were  UNABLE  to  consider  in  the  MODIA  planned  course. 


55.  List  planning  factors  that  you  would  like  to  see  considered  in  a 
simulated  course  operation? 


i 


Please  refer  to  the  Resource  Utilization  Model  output  of  the  course  operatic 
for  the  course(s)  which  you  designed  using  MODIA.  The  following  questions 
refer  directly  to  the  various  summary  information  and  reports  tiiat  make  up 
the  RUH  output.  Indicate  by  placing  a check  mark  in  the  appropriate  space, 
the  value  of  the  information  contained  in  the  report  to  you  as  a course 
planner.  Also  Indicate  whether  you  could  normally  consider  this  type  of 
information  in  revising/planning  a course  by  conventional  means. 

56.  Report  1.  Sunmary  of  course  design 

a.  What  is  the  value  of  this  information  to  you  in  revising/planning 

courses? 


b.  What  is  the  extent  to  which  you  would  normally  consider  this  type 
of  information  in  conventional  course  revision/planning? 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


57.  Report  II.  Resources  Required  and  Capacity  by  Learning  Event. 

a.  What  is  the  value  of  this  information  to  ycu  in  revising  or  planning 

courses? 


1 


NO 

VALUE 


TTTTir 

VALUE 


"RODTrSTT 

VALUE 


VALUABLE”' 


TYTEKhY' 

VALUAbLE 


b.  What  Is  the  extent  to  which  you  would  normally  consider  this  typ*-  of 
Information  Is  conventional  course  revision/planning? 


NEVER — 

CONSIDER 


"SELDOM 


: 1 : : 1 : : 1 : 

Occasionally  frequently  Always- 


c.  How  much  use  could  you  make  of  this  report  In  putting  together  a PO I ? 


• • 


NOSE" 


• • 


“VERY  nTTLE 

USE 


WERATe USEFTT 


: 3 : 

eTTRemel  , 

USEFUL 


58.  Report  III.  Learning  Events  by  Resource  Type 

a.  What  Is  the  value  of  this  information  to  you  in  revising  or  planning  a 
course? 


: 1 : 

NO  "VALUE" 


: 1 : 

TTTTTir 

VALUE 


1 


"ROBERATT 

VALUE 


"VALUABLE 


TxTRemelv 

VALUA6LE 


b.  What  Is  the. extent  to  which  you  would  normally  consider  this  type  of 
Information  In  conventional  course  planning  or  revision? 

1 * No  answer 


• • 


NEVER OTDR- 


1 : 


• • 


. OCCASIONAL!  Y FTOJLfENTLY AUNAVT 
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c.  Howmuchuse  could  you  make  of  this  report  in  putting  tonether  a PO I ? 


59.  RECAP  Report  1.  Report  of  Course  Operation. 

a.  What  is  the  value  of  this  Information  to  you  in  revising  or  planning 
a course? 


: : : 1 : 

NO  VALUE  LITTLE 

VALUE 


:1  : : 1 : : : 

wmm vscumt rnmcTY 

VALUE  VALUABLE 


b.  What  is  the  extent  to  which  you  would  normally  consider  this  type 
of  information  in  conventional  revision  or  planning? 


. 1 . . 1 . 

SEIM  OeCfiSIMILY 


: : : 1 : 

frequently  Always 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a FOI? 


: 3 : 

RZTG5F 


rmtr 

USE 


otrate ijwk 

USE 


EXTrEmKT 

USEFUL 


60.  RECAP  Report  II.  Students  and  Sections  by  Learning  Events. 

a.  What  Is  the  value  of  this  information  to  you  in  revising  or  planning 
a course? 


: : 1 : : 1 : : 1 • • 

no  value  rmii  rotate  vatuabtc — - — extremely 

VALUE  VALUE  VALUE 


-62- 


b.  What  is  the  extent  to  which  you  would  normally  consider  this  type  of 
information  In  conventional  planning  or  revision? 


; ! 1 . . 
OTBIBIWIIY  i RfODERTTr  ~ATwSt:,' 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POi? 


trhtE — 

USE 


UStrUL  " EXTREMELY 
USEFUL 


61-  RECAP  Report  II J.  Student  Queues  by  Learning  Event. 

a.  What  is  the  value  of  this  information  to  you  in  revising  or  planning 
a course?  a 


lllaili 


VALUE 


VALUE 


LV;WW:1I 


VALUABLE 


b.  What  is  the  extent  to  which  you  would  normally  consider  this  type  of 
information  in  conventional  revision  or  planning? 


lags: 


UN*:; 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


im^Eir 

USEFUL 


62.  RECAP  Report  IV.  Resource  Utilization  by  Learning  Event. 

a.  Whet  is  the  value  of  this  information  to  you  in  revising  or  planninq 
a course? 


• • 


wrvAcur 


. 2 . 

trrrir 

VALUE 


1 


HODEMTT 

VALUC 


muABcr 


'EXTREMELY 

VALUABLE 


b.  What  is 

the  extent 

towhichyou  would  normally  consider  this  type  cf 

information 

in  conventional  planning  or  revision? 

• • 

• • 

: 2 ; 

:1  : 

% . 

ritVER 

SELDOM 

OCCASIONALLY 

“Frequently 

TOA ys 

c.  How  much  use  could 

you  make  of  this  report 

in  putting  together  a POI? 

« • 

• * 

: 2 : 

: 1 : 

. . 

. 

Nd  usE 

crrm 

Moderate 

Lisrrur- 

ExTREmely 

USE 

USE 

USEFUL 

63.  RECAP  Report  V. 

Resource  Utilization  by  Resource  Type. 

a.  What  Is 

the  value 

of  this  information  to  you  in  revising  or  planning  a 

course? 

« 

* • 

, , 

: 2 : 

: 1 : 

wtwcoe 

LfTTCT 

MODERATE 

VALUABLE 

EXTREMELY 

VALUE 

VALUE 

VALUABLE 

b.  What  is 

the  extent 

to  which  you  would  non.ially  consider  this  type  of 

information 

in  conventional  revision  or  planning? 

• • 

• • 

: 2 : 

• • 

• • 

n : 

~ -lUlllff 
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c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POP 


: 1 : 


'M5TKA1 

USE 


: 2 : 

uirnu 


'ExTftWir 

USEF'JL 


64‘  into ^MODCOM?  **  >0U  t0  COllect  and  pr<*Pare  f!dta  retired  for  input 


(SPECIFY) 


65*  How  useful  was  the  Information  output  from  MODCOM? 


M 

USEFUL  ‘ 


EXTREMELY 

USEFUL 


66.  How  much  of  the  required  MODCOM  input  did  you  get  from  the  RUM  simulation? 


2 : : l : 

ERY  LITTLE 


67.  How  much  of  the  required  input  did  you  get  from  other  sources? 


• 1 • 

• 1 i 
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68.  List  the  office  symbols  of  the  other  sources  and  estimate  the  amount  of 
the  information  required  by  MQDCGM  that  was  provided  by  the  respective 
sources. 


69.  How  confident  are  you  in  the  course  costs  depicted  by  MODCGM? 


; 2 

: 1 : 

: 1 : 

: 1 : 

: : 

NONE 

VERY 

LITTLE 

SOMEWHAT 

OONFIDIKT 

EXTREMELY 

CONFIDENT 

70.  How  nuch  of  the  information  required  as  input  to  MODOOM  would  normally 
be  available  to  you  (assuming  you  had  no  course  simulation  — that  is,  no 
RUM  output)? 


! 5 i • «•  ••  • • • • ? 1 . ■ ^ ■ * * * • 

15%  20%  30%  J0%  50%  60%  70%  ‘ ~80%  90%  100% 

71.  List  that  information  that  is  required  by  MODCOM,  and  which  can  only 
be  generated  as  a result  of  the  course  simulation  on  the  HJM? 


72.  How  often  would  you  use  MDDCOM,  in  your  opinion,  if  it  were  the  only 
part  of  the  M0DIA  system  available  to  you? 


: 3 : : : : : _ _ : •_  ; 

NEVER  SELDOM  : (XX^SIONALLY  * FTCX&ENTLY  ALWAYS 
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73.  How  would  you  usa  It  (short  sentence  or  two,  please)? 


74.  List  those  offices  and/or  organizations  within  the  technical  training 
wing  which  you  feel  uses  the  kind  of  Information  provided  by  MODCOM. 


75*  those  i offices  and/or  organizations  within  the  technical  school 
which  you  feel  should  use  the  information  provided  by  MODCOM,  or  who 
could  benefit,  from  the  use  of  this  type  of  Information. 


76.  In  your  opinion,  how  should  MDDIA  be  used  (a  short  sentence  or 
two)? 


77.  In  your  opinion,  who  should  use  MDDIA  (Specify  "who  at  each 
organizational  level,  i.e.,  training  evaluation,  plans,  operations 
etc.  — You  can  specify  NONE  or  MORE  THAN  ONE) ? 

Technical  School  Personnel  (Center  level) : 


Technical  Training  Group  personnel: 
Branch  Personnel: 


Hq  ATC  personnel: 

78.  For  each  organizational  level  you  checked,  please,  in  a sentenoe 
or  t»o,  explain  why  they  would  use  MODCA? 

Technical  School: 


Technical  Training  Group: 


Branch: 


Hq  ATC: 


79.  Vtiat  do  you  think  would  be  the  role  of  those  organizations  you 
indicated  in  using  MOOIA? 


80.  Please  list  any  additional  contents  you  care  to  make  about  your 
experience  with  MODIA,  its  usefulness,  or  any  suggestions  you  nay  have 
for  improving  the  system. 
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INSTRUCTIONS  FOR 
ISD  TEAK  QUESTIONNAIRE 


participated  as  members  of  an  ISD  team  in  providing  course 
planning  data  for  tha  NQDIA  interface  team  to  exercise  the  MODIA  system 
in  plraiag  /oar  aouraa.  You  are  in  a position  to  make  a number  of 

aaaoaming  MOOIA  requirements  for  operation,  as  well  as  the 
9~  HDIAsystem  output  for  you  in  planning  course  revisions. 
.y*  «a*er  of  ^pacific  questions  oonceming  details  of  your 

experience  with  the  NODIA  system.  The  judgments  required  of  you  are 
*pp*.v«djaat.*  in  nature,  ixat  nevertheless,  please  exercise  thoughtful 
consideration  of  each  question.  Your  responses  will  be  kept  in  oorplete 
oonriJmma.  Oily  ramary  statistical  results  of  your  responses,  combined 

^ of  other8'  ^11  be  used  in  deciding  on  the  utility 

°f  bhe  NQDIA  ays tun  to  course  planners . 


2.  Plnaao  read  each  question  carefully  and  indicate  your  response 
on  tha  rating  soala  by  placing  a check  mark  BETWEEN  one  of  the  five 
sets  of  dalimitera.  When  you  are  finished,  make  sure  you  have  oonpleted 
infbmation  sheet,  and  that  you  have  put  your  name  in  the 
~*~°***”  place.  Place  the  ocnpleted  questionnaire  in  the  envelope 
provided  and  return  it  to  TIGOTT.  Rsnanber,  no  other  Technical  Training 
wng  peraotnel  will  see  your  responses.  The  inforration  will  be  analyzed 
and  IX— Wail  in  ramary  statistical  form  by  the  Technology  Applications 
Omitar  at  Lackland  AFB,  Texas. 


3.  If  ycu  have  any  additional  comments,  a clarification,  or  explanations 
regarding  wy  particular  questions,  use  the  back  of  the  sheet  containing 
tha  question.  Indicate  the  question  nunber  by  your  comments . 


-70- 


Appendix  5 


GENERAL  INFORMATION  SHEET 


A.  INp^UCTWN:  You  have  participated  In  the  evaluation  of  the 

MOOIA  ^ysteaat  Keesler;  therefore,  you  are  in  the  best  position 
to  mm  a water  of  assessments  concerning  the  usefulness  of  MODIA 
**  1 toe]  flooding  a training  course.  This  historical  sheet  has 
JJ*  ,*??•*  **,1st  y°u  1n  furnishing  us  with  specific  Informa- 

tlod  relative  to  your  background.  Please  answer  each  question  and 
retom  the  history  sheet  to  the  project  monitor. 

B.  iBACMQMUMD: 

1.  Sox:  Mole Female 

2.  Grade/Rank: 

3.  Education:  Years  High  School  Years  Collge 

Degree  (Indicate  Type) 

4.  Group  Membership  

5.  Mhat  Is  your  job/position  title? 

6.  Hom  long 

7.  Experience: 

a.  Training  Course  Planning:  Years 

b.  Instructor:  Years 

c.  Other: 

8.  Total  Years: Federal  Service At  Keesler  AFB 
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1.  How  easy/difficult  was  it  to  assign  objectives  (course  content)  on 

the  User  Interface? 


BTHMELT 

DIFFICULT 


ttobewclv mr 

DIFFICULT 


VfftY 

EASY 


2.  How  easy/difficult  was  it  to  assign  subject  matter  types  to 
learning  objectives? 


ExTftEMELY" 

DIFFICULT 


"DIFFICULT 


: 1 : 

MODERATELY 

DIFFICULT 


EASY 


VERY 

EASY 


3.  How  difficult/easy  was  it  to  assign  learning  event  types? 


extremely 


difficult 


DIFFICULT 


MODERATELY 

DIFFICULT 


EASY 


VERY 

EASY 


♦.  About  how  much  time  did  you  spend  locking  for  the  information  needed 
by  the  User  Interface  Program? 


«mr 

AT  ALL 


: 1 


MOdEWYE  "Amount 
OF  TIME 


:2 


TWTTTdRT 

OF  TIME 


5.  About  how  much  of  the  information  demanded  by  the  User  Interface 
Program  would  be  readily  available  to  you  in  normal  course  revision/ 

planning? 


• 2 


HOST 


1W 

LITTLE 


iBtfUT 

HALF 


MOST 


ALL 
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6.  How  much  of  the  training  content  of  your  course  were  you  able  to 
assign  using  the  User  Interface? 


• • 


• • 


wwr 


tery 

LITTLE 


1 


TOUT TO ST- 

HALF 


■ALT" 


7.  About  how  many  Teaching  Methods  were  you  able  to  specify  using  the 

User  Interface? 


TORT 


ithrr 


FEW 


about 

HALF 


w 


8.  List  all  of  the  teaching  methods  you  were  unable  to  specify 
using  the  User  Interface? 


9.  How  many  of  the  required  resources  were  you  able  to  assign  to  subject 
matter  types? 


TORT 


yjf 


w 


FEW 


"ACT 


HALF 


10.  How  realistic  was  the  assignment  of  resources  (in  your  opinion)  in 
your  course  design? 


TP^LV’ 1 
UNREALISTIC 


• 1 • 


• • 


~m justeiv 

REALISTIC 


: 1 : 

-EXTREMELY 

REALISTIC 
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11.  How  «any  tests  and  reviews  could  you  assign 


NOT  NEARLY 
AS  MANY  AS 
NEEDED 


LESS  THAN 
I WOULD 
HAVE  LIKED 


ADEQUATE" 


ENOUGH 


ENOUGH 


12.  Did  the  User  Interface  provide  you  with  enough  options  for  describing 
student  arrivals? 


NOT  NEARLY 
AS  MANY  AS 
I WOULD  HAVE 
LIKED 


i~EsS  ThAn 
I WOULD  HAVE 
LIKED 


Adequate 


ENOUGH 


IWt  TRAN 

ENOUGH 


13.  Did  the  User  Interface  allow  you  to  break  your  course  into 

enough  increments,  compared  to  conventional  course  revision  and 

planning? 


FEW 


_L 


■ADEQUATE 


Timor 

MORE  THAN 
NEEDED 


imrmr~ 

I WOULD 
NORMALLY  USE 


14.  How  much  time  did  it  take  to  design  alternative  course  plans  after 
the  baseline  course  was  put  in  the  User  Interface? 


1 


A- 'SREAT 
DEAL  OF 
TIME 


MORE  TIME 
THAN  IT  TOOK 
FOR  THE  BASE- 
LINE 


SAME  TIME 


TE5$  TViAT 
THE  BASE- 
LINE 


710  TIME- 

AT  ALL 


15.  About  how  much  of  a deterence  to  designing  alternative  courses  was 
the  time  you  spent  using  the  User  Interface? 


Deterence  i small  rwibmn  xtoioterabix^  great 

AT  ALL  DETERENCE  DETERENCE  DETEf:ENCE  DETERENCE 


16.  How  useful  was  the  User  Interface  program  in  helping  you  consider 
planning  factors  in  revising  and  planning  your  courses? 


AT  ALL 


: 1 : : 1 : r : : • 

VERY  LITTLE  R0TJE5T  CONSIDERABLY  EXTREMELY 

USE  USEFULNESS  USEFUL  USEFUL 


17.  How  many  planning  factors  were  you  able  to  consider  in  designing 
or  revising  a course  using  MODIA? 


; : 2 • • . . 

ONTTA  ATADEQUATE  — R05T~ 

THE  NEEDED  FEW  NUHBER 

FACTORS  COULD 
*E  CONSIDERED 


AIL-FACTOR$ 
I NEEDED 


18.  List  all  other  problems  or  comnents  you  would  like  to  mention  with 
respect  to  the  use  of  the  User  Interface  programs. 


-75- 


22.  How  many  managerial  guidelines  were  you  able  Lo  incorporate  into 
your  alternative  course  design? 


I 


NUMBERS 


23.  How  easy/difficult  was  it  to  incorporate  these  changes  in  alternative 
course  designs? 

1 = No  response 


Fill  iii  — iTTT 


DIFFICULT 


EASY 


24.  How  easy/difficult  was  it  to  use  MODIA  in  simulating  courses  with 

shared  resources? 


: 1 : 

TOT 

DIFFICULT 


DIFFICULTY 


TxTremeLy 

EASY 


1 


25.  How  valuable  do  you  think  MODIA  simulation  will  be  in  revising/ 
planning  courses  with  shared  resources? 


VALUE  VALUE  VALUE 


VALUABLE 


26.  Compared  to  conventional  course  revision/planning,  how  much  time  do 
you  think  you  would  save  using  MODIA? 


1 


27.  If  MODIA  were  a fully  operational  system,  how  much  do  you  feel  it 
would  complicate  the  revision/planning  process? 


w ar 


LITTLE 


"VERY" 

MUCH 


28.  Now  Important  do  you  feel  it  is  to  be  able  to  consider  alternative 

course  designs? 


TOTALLY 

UNIMPORTANT 


"otHry 


LITTLE 

IMPORTANCE 


TO&ATELY LL  ‘ 


IMPORTANT 


very 

IMPORTANT 


"VITALLY 

IMPORTANT 


29.  How  confident  are  you  in  the  results  of  the  simulation  of 

course  operation? 


WT 

CONFIDENT 
AT  ALL 


"VERY 

LITTLE 

CONFIDENCE 


imrnmr 

CONFIDENT 


"VERY 

CONFIDENT 


"EXTREMELY 

CONFIDENT 


30.  How  often  do  you  think  you  would  use  MODIA  In  the  future  were  it  to 
become  fully  operational? 


: 2 ; 


"SOMETIME OFTEN" 


31.  How  easy/difficult  to  interpret  was  the  output  from  the  Resource 
Utilization  Model? 


EXTREMELY 

DIFFICULT 


: 2 • 

"VERY- 


DIFFICULT  DIFFICULT 


: : : 1 : 

"MODERATELY EAST 


"EXTREMELY 

EASY 


8- 


32.  How  useful  for  course  planning  was  the  simulation  output  from  the 
Resource  Utilization  Model? 


Ejn 


USE  AT 
ALL 


: 2 : 


use 


: 1 : 


lUiiTJl 


EXTRCMELV 

USEFUL 


33.  List  the  planning  factors  that  you  would  normally  consider  in  course 
design  which  you  were  UNABLE  to  consider  in  the  MOOIA  planned  course. 


34.  List  planning  factors  that  you  would  like  to  see  considered  in  a 
simulated  course  operation? 


Please  refer  to  the  Resource  Utilization  Model  output  of  the  course  operation 
for  the  course(s)  which  you  designed  using  MOOIA.  The  following  questions 
refer  directly  to  the  various  summary  information  and  reports  that  make  up 
the  RUM  output.  Indicate, by  placing  a check  mark  in  the  appropriate  space, 
the  value  of  the  information  contained  in  the  report  to  you  as  a course 
planner.  Also  Indicate  whether  you  could  normally  consider  this  type  of 
Information  in  revising/planning  a course  by  conventional  means. 

35.  Report  1.  Suwnary  of  course  design 

a.  What  is  the  value  of  this  information  to  you  in  revising/planning 

courses? 

. 2 ■ • 1 • • 

wrwui ctttl? wmm ime nnmcr- 

VALUE  VALUE  VALUABLE 

b.  What  is  the  extent  to  which  you  would  normally  consider  this  type 
of  Information  in  conventional  course  revision/plarming? 


w 


6cc. 


i 


mm to — rreouENUT 


ALUATs" 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


ncust 


: 1 : : 1 : : 1 : 

vm "LiTTie — mmm useful 

USE 


EXTREMELY 

USEFUL 
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36.  Report  II.  Resources  Required  and  Capacity  by  Learning  Event. 

a.  What  is  the  value  of  this  Information  to  you  in  revising  or  planning 
courses?  * 


VALUE 


: 2 : 


VALUE 


VALUE 


I'/JIiLWlj 


VALUABLE 


b.  What  Is  the  extent  to  which  you  would  normally  consider  this  type  of 
information  in  conventional  course  revision/planning? 


CONSIDER 


: 2 : 


kialOTI 


immr1  wm  mini 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


KIiUMs 


n i 


ITLIidlll 


USEFUL 


37.  Report  III.  Learning  Events  by  Resource  Type 

course?*  Va^ue  ^n^ormaTion  to  you  in  revising  or  planning  a 


VALUE 


VALUE 


iv.urj.niia 


VALUABLE 


b.  What  Is  the  extent  to  which  you  would  normally  consider  this  type  of 
Information  in  conventional  course  planning  or  revision? 


. 1 . 


c.  Howmuchuse  could  you  make  of  this  report  in  putting  together  a POI? 


• • 

• • 


1 . 


. 1 s 


NO  USE 


• 1 • 


LITTLE 

USE 


MODERATE 

USE 


USEFUL 


EXTREMELY 

USEFUL 


38.  RECAP  Report  1.  Report  of  Course  Operation. 


a.  What  Is  tho  value  of  this  information  to  you  in  revising  or  planning 

a course? 


wrmw 


: 1 : 

TTTrtr 

VALUE 


WTT 

VALUE 


"EXTREMELY 

VALUABLE 


b.  What  is  the  extent  to  which  you  would  normally  consider  this  type 
of  information  in  conventional  revision  or  planning? 


1 


mw 


TELDOM 


: 2 : : : 

OCCASIONALLY  FREOUENTlY 


ALWAYS 


c.  Hew  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


• • 


OTJ5T 


TTTTCT 

USE 


TWSTT 

USE 


USEFUL 


"EXTREMELY 

USEFUL 


39.  RECAP  Report  II.  Students  and  Sections  by  Learning  Events. 

a.  What  is  the  value  of  this  information  to  you  in  revising  or  planning 
a course? 


b.  What  Is  the  extent  towhichyou  would  normally  consider  this  type  of 
information  in  conventional  planning  or  revision? 


: 1 : 

never" 


: 2 : : : : : 

■OCCASIONALLY — FREtflJEliiTEY ROKyT 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


: 1 : 

i * 

; 7 ; . 

; ; 

n6  Use 

'TITTLE' 

USE 

MOoerate  useful 

USE 

EXTREMELY 

useful 

40.  RECAP  Report  III.  Student  Queues  by  Learning  Event. 

a.  What  is  the  value  of  this  Information  to  you  in  revising 
a course? 

or  planning 

• • 

• • 

: 2 : : 1 : 

. . 

NO  value 

“TITTLE 

VALUE 

MODERATE  VALUABLE 

VALUE 

EXTREMELY 

VALUABLE 

b.  What  is 
information 

the  extent  to  which  you  would  normally  consider 
in  conventional  revision  or  planning? 

this  type  of 

• • 

# • 

• • 

• • 

: 1 : ; 2 . 

NEVER 

SELDOM 

OCCASIONALLY  FREQUENTCY 

A0JA7S 

c.  How  much 

use  could  you  make  of  this  report  in  putting  together  a POI? 

• 

: 2 : : 1; 

# # 

NO  USC 

TITTLE 

USE 

moderate  useful 

use 

EXTREMELY 

USEFUL 

• 
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41  RECAP  Report  IV.  Resource  Utilization  by  Learning  Event. 

a.  Whet  1»  the  value  of  this  information  to  you  in  revising  or  planning 

a course? 


leiail: 


VALUE 


. 2 . 


L'lliiarJ.Via 


IV:lll7:l:lla 


VALUE 


VALUABLE 


b.  What  is  the  extent  towhichyou  would  normally  consider  this  type  of 
Information  in  conventional  planning  or  revision? 


Mai  Mil 


miiMHi  MIAMI  10 IMI 


JS’/AXI 


c.  How  much  use  could  you  make  of  this  report  in  putting  together  a POI? 


mi 


IWluRa 


iimuiii 


:ia:isn 


USEFUL 


42.  RECAP  Report  V.  Resource  Utilization  by  Resource  Type. 

a.  What  is  the  value  of  this  Information  to  you  in  revising  or  planning  a 
course? 


IRIsila 


VALUE 


i:  oiaiLWa 


VALUE 


17.1  H7.1 


VALUABLE 


b.  What  Is  the  extent  to  which  you  would  normally  consider  this  type  of 
Infonaatlon  in  conventional  revision  or  planning? 
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c.  How  much  use  could  you  make  of  this  report  in  puttinq  together  a POI? 


: : : 2;  ; 1 : : ; 

NtrusE  rrmr  hddcrrte  ssefuc  rtstot 

USE  USE  USEFUL 


<3.  How  long  did  It  take  you  to  collect  and  prepare  data  required  for  input 
Into  MODCOM?  30  hrs 

25  hrs 


• 1#  • • • • ** 

i to  3 i to  5 Tto~7  OTHER 

(SPECIFY) 


44.  How  useful  was  the  Information  output  from  MODCOM? 


: : : 1 • • 1 • • 1 

ffirosE  mr  nme  wmmis  imw extremely 

USE  USEFUL  USEFUL 


45.  How  much  of  the  required  MODCOM  input  did  you  get  from  the  RUM  simulation? 


' i • ••  • C • ••  . i 

Son!  very  little  §omT  fiosf  in: 


46.  How  much  of  the  required  input  did  you  get  from  other  sources? 


VERY ’LittLe biT 


: 2 : 

M(3sT" 


; 1 : 

Wi~ 
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47.  List  the  office  symbols  of  the  other  sources  and  estimate  the  amount  of 
the  information  required  by  MODCOM  that  was  provided  by  the  respective 
sources. 


41.  How  confident  are  you  In  the  course  costs  depicted  by  MODCOM? 


• • • • 

vert  tmmr 

LITTLE 


• • 0 • 

tONFiBEMT EXTREMELY 

CONFIDENT 


•9.  How  much  of  the  Information  required  as  input  to  MODCOM  would  normally  be 
available  to  you  (assuming  you  had  no  course  simulation  —that  is.  no 
RUM  output)? 


••  * * 'X  ! ! X;  ; ; ;;  ..  . . . , " X * 

. 101  201  30%  40%  50%  60%  70%  80%  90%  100% 


50.  List  that  Information  that  Is  required  by  MODCOM,  and  which  can  only  be 
generated  as  a result  of  the  course  simulation  on  the  RUM? 


51.  How  often  would  you  use  MODCOM,  in  your  opinion,  if  it  were  the  only  part 
of  the  MODIA  system  available  to  you?  — *■ 


. 2 ■ . i . 

■5ECDW  Occasionally  FRRJOenTlY WSm 
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52. How  would  you  use  it  (short  sentence  or  two,  please)? 


53. List  those  offices  and/or  organizations  within  the  technical  training 
wing  which  you  feel  uses  the  kind  of  information  provided  by  MODCOM. 


! 


54.  List  those  offices  and/or  organizations  within  the  technical  school 
which  you  feel  should  use  the  information  provided  by  MODCOM,  or  which 
could  benefit  from  the  use  of  this  type  of  information. 
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55.  In  your  opinion,  hew  should  MOO  LA  be  used  (a  short  sentence  or 
two)? 


56.  In  your  opinion,  who  should  use  MODIA  (Specify  "who"  at  each 
organizational  level,  i.e.,  training  evaluation,  plans,  operations 
etc.  - You  can  specify  NONE  or  MORE  THAN  ONE)? 

Technical  Training  School  Personnel  (Center  Level) : 

Technical  Training  Group  personnel : 

Branch  personnel :_ 

Hq  ATC  personnel: 

57.  Ftor  each  organizational  level  you  checked,  please,  in  a sentence 
or  two,  explain  why  they  would  use  MX  LA? 

Technical  School: 


Technical  Training  Group: 


Branch: 


Hq  ATC: 
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58.  What  do  you  think  would  be  the  role  of  those  organizations  you 
indicated  in  using  MDDIA? 


59.  Please  list  any  additional  ocntnen ts  you  care  to  make  about  your 
experience  with  MODJA,  its  usefulness,  or  any  suggestions  you  may 
have  for  inproving  the  system. 
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INSTRUCTIONS  FOR  THE 
TRAINING  MANAGERS'  QUESTIONNAIRE 


1.  As  a manager  of  various  aspects  of  technical  training  course  opera- 
tion, you  have  bean  asked  to  participate  in  the  evaluation  of  the  MODIA 
planning  qyetasi.  You  are  in  a position  to  make  a metier  of  assesanents 
concerning  the  usefulness  of  MODIA  products  to  the  effective  management 
of  training  course  operations,  and  to  MDDIA' s usefulness  to  the  training 
manager  as  a planning  tool.  In  the  following  pages,  you  will  find  a 
nuriaer  of  specific  questions  concerning  your  experience  with  MDDIA  and 
your  opinions  as  to  its  potential  usefulness.  The  judgments  required  of 
you  ere  appresdnate  in  nature,  but  please  exercise  thoughtful  consideration 
for  each  question.  Only  stannary  statistical  results  of  your  responses 
oonbinsd  with  responses  of  others  will  be  used  in  deciding  on  the  utility 
of  the  MDDIA  system. 

2.  Please  read  each  question  carefully  and  indicate  your  response  on 
the  rating  acala  by  placing  a check  mark  in  the  appropriate  space. 

Wwn  you  are  finished,  make  sure  you  have  oonpleted  the  general  information 
sheet,  and  that  you  have  put  your  name  in  the  indicated  place.  Place  the 
collated  questionnaire  in  the  envelope  provided  and  return  it  to  TTOOrr. 
Vmnmtmr,  no  other  Technical  Training  Wing  personnel  will  see  your  responses. 
The  information  will  be  analyzed  and  presented  in  stannary  statistical  form 
by  the  Technology  Applications  Center.  If  you  have  additional  cements, 
clarification  and/or  explanations,  regarding  any  particular  question, 
pleeee  make  than  on  the  beck  of  the  sheet  containing  the  question.  Please 
indicate  the  question  number. 


Appendix  6 


A 


questionnaire  for  training  managers 


h sirailaticn  to  you  as  a training  nanaoer, 

in  spotting  potential  problem  in  course  operation? 


: 1 : 

NO  USE 
XT  ALL 


: 5 i 

TUu  Lima 

USB 


: 1 : 

MXERATC 

USEFULNESS 


-•>  : 2 ; ; . 

CONSIDERABLY  kXTRf-MPTV 

USEFUL  USEFUL 


plmm  list  the  potential  problems  that  MODIA  allowed  you  to  spot. 

- problems  In  instructor  manning  in  lab  situations 

- Instructional  sequencing 

- Number  of  required  classrooms 

- No  problems  not  already  foreseen 

- Student  bottlenecks 


2.  Were  any  of  the  problems  depicted  in  the 
you  vrould  have  foreseen  without  MCDIA? 


MOD  LA  simulation  problems 


8 Yes  1 No 


V"  your  response  was  yes,  please  list  those  problems  you  could  haw 
^oreseen  without  MODIA  simulation  and  explain  how  you  would  haw  foreseen 


- Manpower  Utilization 

- Delays  in  student  progress 

- Costs  of  Training 

- Laboratory  Utilization 


^di  p^leme  vouici  you  have  been  unable  to  foresee  without  the  MODIA 
simulation?  (Please  list) 

- None 

- }]]  Problems  were  known  before  MODIA  simulation 

-All  of  the  problems  established  by  MODIA  programs  were  foreseen 
and  attacked  without  MODIA 
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3.  How  realistic  were  the  alternative  course  designs  provided  you  by 
your  ISD  team  member? 


: 3: 


UNREALISTIC  REALISTIC 


• 2 • 

• *-  • 


MODERATELY 

REALISTIC 


2 


VERY 
REALISTIC 


VIVIDLY 
REALISTIC 


P1*M*  oannent  an  aspects  of  the  alternatives  which  you  feel  were  helpful 
or  unrealistic: 

- The  alternatives  could  only  provide  single  shift  simulation  due  to  amount  of 

inputs  required. 

- MODIA  was  limited  because  suggested  alternatives  could  not  be  used  because  of 

the  250  learning  event  restriction. 

- The  alternatives  that  were  used  indicated  the  results  that  were  anticipated. 


4.  Hew  nuch  time  did  it  take  for  the  ISD  team  to  generate  alternative 
course  designs  for  you? 


NO  TIME 
AT  ALL 


LITTLE 


: 4 . 


AMOUNT 


. 4 . 
• • 


CONSIDERABLE 


AMOUNT  OF  TIME 


5.  Hew  many  of  the  course  design  changes  you  recomended  were  the  ISD 
interface  team  members  able  to  incorporate  into  the  alternative  course 
designs? 


Lvi  foi 


Do  you  have  any  additional  oerments  on  the  alternatives  the  ISD  teams 
designed  on  MODIA  far  you? 


6.  Could  you  understand  the  output  of  the  MODIA  simulation? 

8 - Yes 
1 - Not  at  all 


Contents? 

7.  Did  the  ISD  tarn  member  explain  the  output  to  you? 
8 Yea  1 No 


8.  Did  you  feel  that  you  could  understand  the  simulated  course  operation 
after  it  was  explained  to  you? 


8 Yes 


1 No 


Omenta? 


9.  In  your  opinion,  would  the  MDDIA  simulation  of  course  operation 
and  course  cost  enable  you  to  better  manage  course  problems  and 
resources? 


: 1: 

VERY  NUCH 
SO 


&XEMHAT 

BETTOR 


2: 


VERY 

LITTLE 


NOT  AT 
All. 


Omenta? 

The  simulation  would  enable  us  to  find  bottlenecks  and  queuing  problems 
before  they  occurred.  It  would  be  extremely  valuable  as  a course  planning 
tool  if  we  were  able  to  program  the  Inputs  for  a two  shift  course. 

Program  needs  expanding  to  allow  other  management  factors  to  be  considered: 
l.e.,  class  schedules,  washback  related  problems  for  rescheduling,  etc. 
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10.  Are  there  any  changes  you  would  like  to  see  made  in  either  the 
aourae  simulation  or  aost  information  that  would  make  the  MODIA  system 
output  more  useful  to  you  as  a course  manager?  If  there  are  any  changes, 
please  list  thsn  and  explain? 

- Computer  time  needs  to  be  increased. 

- Limits  on  the  number  of  inputs  requires  increasing 

- Output  needs  to  be  reorganized  by  higher  Hq  as  valid  tool  for  increasing 

or  decreasing  manning  and/or  facilities. 

- Cost  data  was  very  difficult  gathering  and  validating. 

- Increase  the  250  training  event  limitation 

- Increase  type  of  learning  events,  teaching  formats  and  teaching  agents. 


11.  How  often  do  you  feel  you  would  use  MODIA  were  it  to  be  adopted  as 
a fully  operational  system? 

; : : 4 : : 3 : : 2 ; !__J 

NEVER  SBJXH  SOMETIME  OFTEN  OCNSTAOTLY 


12.  Overall,  hew  valuable  would  MODIA  be  to  you  in  planning  a course 
revision? 


NO  VALUE 


• I • 

LITHE  MODERATE 

VALUE  VALUE 


• 2 • 
VALUABLE 


EXTREMELY 

VALUABLE 


Additional  Garments? 


13.  How  confident  are  you  in  the  results  of  the  simulation  of  course 
operation? 


• • 
S5f 


OONFTDOfT 
AT  ALL 


: 5 : : 2 : 

VETO  MODERATELY 


UTILE  CONFIDENT 

CONFIDENCE 


• 2 • • • 

» **  « » • 

VERY  EXTREMELY 

CONFIDENT  CONFIDENT 


I 


17.  Do  you  feel  the  simulation  of  alternative  designs  could  be  of 
value  to  you  as  a oourse  manager? 

5 Yes  4 No 


18.  In  *4>at  ways  wxild  you  use  information  provided  by  the  cost  nodel? 


19.  What  was  the  cost  relationship  of  the  baseline  course  to  the 
alternatives  you  asked  the  ISD  team  members  to  plan  on  MDDIA? 


Altamative  1: 

1 

a.  Much  more  expensive 

1 

b.  More  expensive 

4 

c.  About  the  same 

3 = No  response 

% 

d.  Less  expensive 

e.  much  less  expensive 

Alternative  2: 

a.  Much  more  expensive 

1 

b.  More  expensive 

3 

c.  About  the  same 

d.  Less  expensive 

3 = No  response 

_e.  Much  less  expensive 
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Cements  or  oqplanations? 


20.  Wire  the  alternative  course  designs  workable  — that  is,  did  they 
aonftrai  to  Air  Training  amend  and  Technical  School  policy? 

1 a.  Ccrpletely  workable 

A 

* b.  Workable  with  minor  changes 

2 c.  Somewhat  unworkable  - major  changes  requirec 


2 * No  response  d.  Totally  unworkable 

Omcnts? 


21.  In  your  cpninion,  how  should  MODIA  be  used  (a  short  sentence  or 
two)? 


I 

* 

22.  In  your  opinion,  who  should  use  MX>IA  (Specify  "who"  at  each 
organizational  level,  i.e. , training  evaluation,  plans,  operations 
etc.  — You  can  specify  NONE  or  MDRE  THAN  ONE)  ? 

Technical  School  Personnel  (Center  Level)  s 4 

Technical  Training  Grotq?  personnel:  _3 

®ran^1  Personnel:  3 course  planners/curricula  

Hg  ATC  personnel:  4 

2 - No  one  should  use  It. 


23.  POr  Mch  organizational  level  you  checked,  please,  in  a sentence 
or  two,  explain  shy  they  vould  use  MXIA? 

Technical  School: 


Technical  Training  Group: 


Branch: 


Hq  ATC: 


24.  What  do  you  tliink  would  be  the  role  of  those  organizations  you 
indicated  in  using  MCDIA? 
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25.  PUase  list  any  additional  oomnants  you  care  to  make  about  your 
cxpariano*  with  MDOIA,  its  usefulness  * or  any  suggestions  you  may  have 
four  inproving  the  system. 

- Increase  program  limits  on  teaching  formats  and  agents. 

- Increase  number  of  learning  events 

» . - Improve  cost  model  to  permit  Insertion  of  other  course  cost.  One  weak 

v area  encountered  Is  In  expendable  supplies.  Our  training  courses  use 

materials  that  are  costly  In  supporting  performance  training  In  the 
laboratories. 

- The  system  must  be  expanded  to  be  worthwhile. 

- The  programs  need  to  be  expanded. 


